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Introduction 


Numerous countries suffer from a shortage of 
technicians and skilled workers, particularly in the 
STEM fields (science, technology, engineering, and 
mathematics), due to a mismatch between the skills 
and interests of the students graduating from or 
leaving the current education system and the needs 
of the labor market. Often, parents and students 
place a high priority on entering and completing 
university, and as a consequence, many students 
pursue academic education in secondary schools 

in order to gain entrance to university, only to find 
themselves entering the job market lacking the skills 
they need for employment and advancement. Further, 
in some countries, the vocational school system is not 
well-respected by the public and focuses heavily on 
preparation of youth for specific jobs in one firm rather 
than preparing them for careers within an industry 

as a whole. This literature review aims to synthesize 
the research evidence about the effectiveness of 
various strategies used by national governments, 
non-governmental organizations, technical schools, 
and industries to strengthen both the quality of the 
technical workforce, as well as the avenues through 
which individuals can access career and technical 
programming. 


To help policymakers develop strategies to meet 

these challenges, we examine research findings in 

two broad solution domains making up the two major 
chapters of this review: Chapter I. Skills Develooment 
and Chapter Il. Provision of Information & Incentives. 
Within each domain, multiple strategies have been 
identified that address these workforce development 
issues by providing opportunities for different population 
groups. While Chapter | focuses on strengthening skills 
development through various types of career and 
technical education, Chapter Il highlights strategies 

to increase access to this type of programming. This 
reviews aims to be international in nature, though some 
workforce development strategies (and thus research 
on these strategies) tend to be region-specific. For 
example, while vocational education research has been 
conducted worldwide, apprenticeship research is largely 
limited to Europe, and cooperative education as well as 
career academy research is limited to North America. 


The five primary strategies assessed in the first chapter of 
this review include various types of career and technical 
education: vocational education, apprenticeships, 


Strategies for Strengthening the Technical Workforce 


cooperative education, career academies, and 
e-learning programs for the workforce. While 
vocational education programs, apprenticeships, and 
e-learning programs exist both within and outside of 
the formal education system, cooperative education 
programs and career academies exist largely within 
the education system. As most programs under 

study are relatively small and region-specific, the 
effectiveness of these strategies is assessed in terms 

of individual-level outcomes including academic 
outcomes (performance, retention, completion, etc.), 
as well as labor market outcomes (employment status, 
wages, length of employment search, etc.), though 
studies on vocational education and apprenticeship 
strategies do include macro-economic-level outcomes 
such as country-level employment rates, job turnover, 
or labor market productivity. These five strategies are 
summarized below: 


Vocational education aims to impart students with 

a certain type of knowledge, understanding, and 
competencies that are necessary for future work 
within a particular industry or occupational field. Its 
occupation-specific curriculum prepares students for a 
specific trade, craft, or career by directly developing 
essential expertise, applicable techniques, and skills for 
employment, often through the use of a work-based 
learning. Vocational education can be offered within 
the formal education system as well as outside of the 
system (e€.g., labor market training programs). 


Bridging the gap between theory and practice, 
apprenticeships provide individuals with an opportunity 
to gain a mix of classroom or workshop instruction and 
on-the-job training. During the course of between 
three to six years (depending on the program), 
apprentices gain real life experience by working under 
the supervision of an advanced professional. 


Similarly, cooperative education (“coop”) is another 
method of combining classroom-based education 
with practical work experience. Coop education is 

a type of internship program that enables students 
to receive career training (typically with pay) for 
academic credit. Coop programs provide individuals 
with an opportunity to explore career choices and to 
network with professionals. 


Additionally, career academies are formal, high school 
based small learning communities organized around 
such themes as health sciences, law, engineering, 

and business and finance. In academies, students 
receive academic instruction at their theme-focused 
high school combined with work-based learning 
opportunities at an industry center, or technical school 
or college. 
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Lastly, e-learning programs involve the use of a 

wide variety of applications and processes such as 
computer or electronic technologies to deliver a 
broad range of educational curricula; these programs 
can target both students as well as individuals already 
in the workforce. Potential benefits of e-learning 
include its ability to provide distance learning (as 
opposed to learning on a campus); to provide a 
blended classroom approach (combination of 
face-to-face and technology-based teaching and 
learning); and to provide an additional avenue with 
which to support teaching and learning practice. 


In addition to improving the skills of the workforce, 
providing access to information and employment 
services in the technology field, as well as monetary 
incentives to students to enter into these fields 

can have a large impact on individual choice 

and subsequently overall access to career and 
technical programming. The provision of information 
and incentives includes three primary strategies: 
informational interventions to potential students, 
strengthening of both informal and formal career 
counseling and employment services, and monetary 
incentives for students. The effectiveness of these 
programs is generally assessed through outcomes such 
as increased enrollment in a particular school or field, 
academic performance in the program of choice, as 
well as various labor market outcomes of those taking 
part in these initiatives. A brief summary of each type 
of strategy is detailed below: 


e Concerns about the low visibility and take-up rates 
of government supported workforce development 
programs have prompted calls to provide more 
information interventions. Information interventions 
include efforts to reach out to students and families 
to educate them about the availability of program, 
the financial returns or the benefits of a certain 
type or level of education, their post-secondary 
aid eligibility, and school performance rankings. 
Information interventions can also include efforts 
to simplify or to provide greater assistance with the 
application process. 


e With more and more diverse career options and 
professional opportunities emerging, career 
counseling and employment services help 
individuals make the right choice about their career 
paths, career development, and career change. 
Career counseling involves the counseling or 
mentoring/coaching of individuals (adults as well 
as adolescents or students) to help them make 
informed career choices and define their career 
paths. This type of counseling can be offered 
formally through schools or other government 
institutions, as well as through informal or private 
channels. In addition, employment services are 
agencies that attempt to match the employment 


needs of an employer with an individual who 
obtains the required skillset and interests. 


¢ Monetary incentives are financial rewards given to 
employees or students to motivate participation or 
performance. With respect to education, monetary 
incentive structures (scholarships, vouchers, 
stipends, etc.) are offered to students to encourage 
enrollment, attendance, and completion; reduce 
dropout rates; and to improve learning outcomes 
and post-secondary school aspirations. 


These three approaches to tackling the shortage 
of technicians and skilled workers apply not only to 
an immediate plan to recruit employees, but also 
to a long-term plan to build and retain a technical 
workforce for the future. 


Our methodological approach to this review was 
systematic; that is, we attempted to examine all of the 
relevant and rigorous empirical research in each of the 
solution domains, following the Cochrane Handbook 
for Systematic Reviews of Interventions (Higgins & 
Green, 2011). The inclusion criteria for the review 
focused on the two main solution domains as well as 
the rigor of the methodology used. Preference was 
given to efficacy trials that used experimental methods 
(randomized controlled trials) or quasi-experimental 
methods (propensity score matching, instrumental 
variables, regression discontinuity designs, difference- 
in-difference designs, natural experiments, time 

series analysis, etc.). This is due to the fact that these 
methods allow for a causal interpretation of the impact 
of each program; that is, the impacts measured are 
fully attributable to the program or policy itself and 

not to other confounding factors such as student 
background variables or the presence of other 
concurrent policies or economic factors. However, 
while these studies have high internal validity (lack of 
bias) in their impact estimates, their external validity 
(the extent to which their findings can be extrapolated 
to other contexts) must be determined case by case. In 
addition, certain high-quality correlational, descriptive, 
and observational studies were included when highly 
relevant, and in some cases policy reports and case 
studies were used to add another perspective to 

the evidence. Note that in addition to the efficacy 
literature upon which the bulk of this paper is based, 
Appendix B provides an overview of relevant and 
recent policy-relevant reports, manuals, and books 
which were identified through our systematic search. 


While the search criteria and search terms differed 
slightly by topic, the studies included in this review 


1 http://www.3ieimpact.org/evidence/ 
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were located in three main ways: (1) searches of the 
following databases: Education Research Complete, 
Business Source Complete, ERIC, Education Full Text, 
Social Sciences Full Text, and EconLit with Full Text, as 
well as Google Scholar; (2) searches of online impact 
evaluation databases such as the International 
Initiative for Impact Evaluation (3ie) Database,’ the 
World Bank Education Database,? and the Youth 
Employment Inventory;? (3) searches of relevant 
international organization websites, including the World 
Bank, UNESCO, OECD, IZA, and ILO; and (4) “citation 
crawling” of relevant meta-analyses and other well- 
cited studies within each field. Appendix C provides 
a detailed description of the search strategy used 
including search terms for each topic. 


Analytically, this review attempts to synthesize the 
literature in such a way that broad patterns in program 
outcomes can be identified, and to the extent that the 
findings of program effects are conflicting, we have 
attempted to explain and learn from this variation. 
Note that this review does not pool effect sizes as 

in a meta-analysis, as the outcomes across these 
interventions differed too greatly to be combined into 
one average measure (i.e., the effects of vocational 
programs were measured in a variety of ways including 
starting salaries, current wages, hours worked, 

time to employment, employment status, formality 

of employment, and educational outcomes (in 
vocational school) including academic performance, 
retention, and completion). 


For each strategy examined below, we provide a 
description of the literature (the population samples 
targeted, the publication types within the literature, 
the outcomes examined across studies, methodologies 
used, and geographic distribution of the studies); a 
summary of the findings; and a brief description of the 
limitations and implications of the synthesis. This review 
is structured as follows: Chapter I. Skills Development, 
details the evidence on the effectiveness of various 
types of career and technical education to improve 
the quality of technical education (this includes 
vocational education and training, apprenticeship 
trainings, cooperative education, career academies, 
and workforce training through e-learning), while 
Chapter Il. Provision of Information & Incentives, speaks 
to the impact of informational interventions, career 
counseling and employment services, and monetary 
incentives in strengthening access to technical 
education. 


2 http://datatopics.worldbank.org/EdStatsApps/Edu%20Evaluation/evaluationHome.aspx?sD=E 


3 http://www.youth-employment-inventory.org/ 
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This chapter addresses the “Skills Development” 
solution domain and includes five main strategies for 
strengthening the career and technical education 
sector and addressing skills shortages: vocational 
education (formal and non-formal), apprenticeships, 
cooperative education, career academies, and 
e-learning programs for the workforce. The section on 
vocational education below details the importance 
of links to industries and the quality of vocational 
training, discusses the macro and micro implications 
of interventions in this field, describes the differential 
effects of programs by gender, and addresses returns to 
public versus private sector vocational programming. 
Secondly, the research on apprenticeship programs 
included in this review addresses an array of topics 
including the country-level factors influencing 
apprenticeship provision, how apprenticeships 
impact wages, transition to employment and length 
of employment, the effect of apprenticeships on 

the macro economy, and heterogeneous effects 
associated with apprenticeships. 


Overall, the studies on cooperative education at the 
secondary and post-secondary level examine the 
effect of coop education on completion and retention 
rates (for both school and work-related outcomes), 

as well as the psychological impact of taking part in 

a cooperative program. Efficacy literature on career 
academies explore their impact on secondary school 
outcomes, post-secondary school outcomes, and 
labor market outcomes. Finally, the e-learning studies 
included in this section explore the lack of significant 
achievement differences between e-learning and non- 
e-learning programs and the heterogeneity that exists 
within these findings. The efficacy literature for each of 
these five approaches is synthesized in greater detail 
below. 
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Vocational Education 


Katharine Conn 


Vocational education refers to education or training 
which is focused on preparation for work in a 
particular industry or field and is often accredited 
or regulated at either the regional or national 

level. Vocational education can be offered both 
within and outside of the education system. Formal 
vocational education programs are a common form 
of secondary or post-secondary education in many 
countries and are often offered within universities, 
community colleges, institutes of technology, 

or other registered training organizations. These 
programs often have a work-based learning 
component. Further, non-formal vocational 
education programs are also very common 
worldwide as a form of “workforce retraining.” In 
some countries the vocational sector interacts with 
established apprenticeship programs (discussed in 
the following section). In this section we describe 
what is known from the research literature regarding 
the efficacy of vocational education programs. This 
section includes a description of the literature as 

a whole (program population targets, publication 
types, outcomes, methodologies, regions covered, 
etc.), the relevant findings associated with this body 
of work, and any limitations or implications of these 
studies. 


Literature Description 


Thirty-six studies of the impact of vocational education 
on youth were identified for inclusion in this synthesis.* 
The following section provides an overview of the 
characteristics of the studies found through the 
systematic review process and which are included in 
this synthesis. 


Study Population. Of the 36 high-quality vocational 
education studies identified for this synthesis, only one 
of them examined an intervention targeted at students 
currently in school [Bettinger, Kremer, and Saavedra 
(2010) focused on Kenyan 5th grade students as they 
chose between the general or a pre-vocational track]; 
the majority of the remaining interventions targeted 
unemployed or low-income youth under 30. 


Publication Type. Of the 36 high-quality vocational 
education studies identified for this synthesis, 15 are 


4 See Appendix C for a description of the search strategy and search terms. 


journal articles, 3 are reports, and the remaining 19 are 
working papers. 


Outcomes. Most of the outcomes examined in the 
vocational education literature focused on the 
individual returns to vocational schooling (wages, 
hours worked, employment status, formality of 
employment, academic performance, retention, 
and completion rates). Only one study (Dmitrijeva, 
2009) examined the impact of a vocational program 
on macroeconomic indicators (aggregate outflows 
from unemployment to jobs). This is largely due to 
the fact that these programs were often too small for 
researchers to expect a general equilibrium effect. 


Methodology. Of these vocational education-related 
studies, 10 employed randomized controlled trials, 

18 employed some form of matching (namely 
propensity score matching), 2 employed instrumental 
variables, 1 used a natural experiment, 2 employed 
Heckman selection correction,® 1 used a regression 
discontinuity design, 1 involved a bivariate duration 
model, and 1 employed multiple strategies 
(randomized controlled trial as well as matching). 


Region. With regard to geographic location, the 
efficacy studies in this sample are from multiple 
continents: three studies in Africa (Kenya, Liberia, 

and Kenya & Zambia); four in Asia (Bhutan, Thailand, 
and two in China); five in Central America and the 
Caribbean (Panama, two in the Dominican Republic, 
and two in Mexico); nine in South America (two in Peru, 
three in Argentina, three in Colombia, and one that is 
pan-Latin American); eight in Eastern Europe/ Western 
Asia (Slovenia, Bulgaria, Latvia, Romania, two in Russia, 
and two in Turkey); five in Western Europe/ Nordic 
countries (Finland, two in Sweden, and two in Norway); 
and two in North America (United States). No studies 
meeting the criteria were found in countries with 
strong reputations for providing effective vocational 
education such as Germany, Singapore, and Taiwan. 


Training field. The majority of these impact evaluations 
did not report detailed information as to the type of 
vocational education offered. Eight studies reported 
that the intervention placed students with “firms,” 

but did not state exactly what kind of firms or skills the 
students gained. One evaluation (Alzua, Nahirnak, 

& Alvarez de Toledo, 2007) of an Argentine program 
“Entra 21” described in detail the kind of ICT training 
its students received. JOvenes en Accidn in Colombia 
(Attanasio, Kugler, & Meghir, 2011) provided training 

in multiple fields: administrative occupations, manual 
occupations, and skilled occupations (IT specialists, 
data entry, surveyors, and accountant assistants). A 
“laid-off” worker re-training program in China provided 
IT and services training. A rural skills development 
program in Bhutan (Chun & Watanabe 2011) provided 


5 Heckman correction is not a quasi-experimental strategy, but it uses two-stage estimation method to correct for selection bias. 
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specific construction skills to participants (carpentry, 
masonry, plumbing, and electrical wiring). And 
Bettinger, Kremer, and Saavedra (2010) evaluated a 
vocational education program in the services and 
manufacturing sector in Colombia. 


Training length/type. The vast majority of programs 
evaluated in this literature were active labor market 
training (non-formal education) programs (32 

studies), followed by evaluations of formal vocational 
education programs (4 studies), and finally, there was 
one program that examined the impact of vocational 
training for current employees. The modal length of the 
training programs in this synthesis was approximately 

6 months (the range is from 3 months to 2 years). All 

of the vocational education interventions evaluated 
here offered students on-site training or workplace 
internships. Approximately 75% of these programs split 
the student's time between coursework and on-the-job 
training. The remaining 25% offered exclusively on-the- 
job training. 


Findings 


The findings from this review detail both the 
effectiveness of these programs in improving 
participant outcomes, as well as the conditions under 
which vocational education programs are most 
successful. The findings emphasize the importance of 
links to industries and the quality of training; discuss the 
macro and micro findings of these studies; describe the 
differential effects of programs by gender; and address 
returns to public versus private sector vocational 
programming. 


Links to industries. There is suggestive evidence that 
interventions which link vocational education and 
training most closely to targeted industries tend to 
be the most successful in terms of student outcomes. 
Programs in which the student receives a paid 
internship are particularly useful in that the employer 
can use the internship as a screening process for 
hiring (given that they have already invested in the 
training of the individual). If students’ wages are not 
paid by the firm, however, this can create perverse 
incentives. To this point, Card, lbarraran, Regalia, 
Rosas, and Soares (2011), in an evaluation of a youth 
training program in the Dominican Republic, found that 
because the interns’ wage costs were fully subsidized 
by the program, employers had a strong incentive 

to fill their slots with new trainees once the subsidy 
period came to an end rather than to hire graduates 
from the program. Further, Chun and Watanabe 
(2011) studied a rural skills development program 

in Bhutan that provided training to participants in 
carpentry, masonry, plumbing, and electrical wiring. 
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But because this training was not linked to an industry 
or firm, the employment benefits were lower than 
expected, causing the authors to conclude that 

there is a need for “a greater emphasis on creating a 
mechanism to connect the training program to income 
generating opportunities via job placement services, 
entrepreneurship, or mentoring services—especially in 
competitive labor markets where there are too many 
trainees in relation to the population” (p. v). 


Innovations in the way in which industries can be 
linked to vocational training programs include an 
implementing organization that also functions as a 
placement office for the unemployed, as in the case 
of the Probecat training program in Mexico. In this 
case, post-training, 70% of the trainees were hired by 
the firm with which they were interning; the remaining 
students then tried to find a position through the SNE 
(Sistema Nacional de Empleo) placement office 
(Delajara, Freije, & Soloaga, 2006). Another example is 
the program Entra 21 in Argentina, which required the 
implementing NGO to commit to inserting 40% of their 
graduates in the labor market (AlzUa et al., 2007). 


Quality of training. Chong and Galdo (2006) found 

in their study of a youth training program in Peru 
(PROJOVEN) that the quality of the training program 
was highly correlated with student success. They found 
that the quality premium was over 20 percentage 
points (in wages) between individuals who attended 
high- and low-quality courses. In their measure of 
quality, they included six different categories: class size, 
expenditures per trainee, eight teacher variables, six 
infrastructure and equipment physical characteristics, 
nineteen curricular structure variables, and nine 
variables characterizing the link between the content 
of the training courses and the institution's knowledge 
about workers and occupational analysis of labor 
demand. 


To ensure the participation of high quality training 
programs, the selection of training institutions is often 
based on a public bidding process that can include 
various eligibility criteria: legal status (formality) and 
the existence of some acceptable level of human 
resources and infrastructure (PROJOVEN in Peru, Chong 
& Galdo, 2006); legal registration, economic solvency, 
quality of teaching, and ability to place trainees after 
the classroom phase into internships with registered 
employers (JOvenes en Accién in Colombia, Attanasio 
et al., 2011); and existence of a written commitment 
by private firms to provide paid internships, past 
training experience, administrative capacity, and the 
adequacy of the courses provided (Projoven in Peru, 
Diaz, & Jaramillo, 2006). 
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Macro findings. Only one quasi-experimental study 
examined the impact of a vocational education 
program on outcomes beyond the individual; 
Dmitrijeva (2009) found that the effects of publically 
provided training programs (namely vocational 
education) for the unemployed in Latvia (part of the 
government’s Active Labor Market Policy) on outflows 
from unemployment to employment was positive and 
statistically significant. Further, the cost-benefit analysis 
findings held that the costs of training were easily 
covered at the aggregate level if the average job 
tenure in the economy approached nine months. 


Micro findings. Again, the vast majority of programs 
evaluated in this literature were active labor market 
training (non-formal) programs (32 studies), followed by 
evaluations of formal vocational education programs 
(4 studies), and a program which examined the impact 
of vocational training for current employees (1 study). 
Overall, results in terms of individual employment, 
earnings, job formality, and other outcomes were 
mixed. Of the 32 studies examining the impact 

of public vocational training on these individual 
outcomes, the majority of these studies evidenced 
either null or statistically insignificant (though slightly 
positive) results (20 studies); the remaining studies had 
either positive (6 studies) or mixed results (6 studies). 
Studies with mixed results included those that were 
positive for employment but negative for wages or 
they were positive for only certain subgroups within 

the larger sample (by age, gender etc.). Further, there 
were unfortunately no clear patterns in the data that 
indicated the presence of an intervention-specific 
moderator. For example, there were no obvious 
patterns between particular program components 

or program length and resulting labor market 
outcomes. Thus, while these results varied due to the 
implementation quality of the interventions themselves, 
they were also likely to vary due to the economic 
context of each country or region. Betcherman, Dar, 
Luinstra, and Ogawa (2000) in a synthesis of active 
labor market training programs found the effectiveness 
of programs to be heavily reliant on the business cycle 
and to be much more successful in times of economic 
expansion. Additional moderating variables could 
include market saturation of certain skilled workers, 
unemployment rates, or the size of the skill-gap, etc. 
For example, in Alzua et al. (2007), the vocational 
program implemented was identical (in principle) 
across countries, but the results differed wildly between 
Argentina and Brazil. In Argentina, there was a positive 
but insignificant effect on employment, a positive 
impact on income (though this decreased over time), 
and no statistically significant effect on formality, 

while in Brazil the program had no significant impact 
on either employment or earnings. One pattern in the 
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data that does clearly emerge, however, is the strong 
impact of these programs on females (discussed in the 
next section). 


Unlike the programs discussed above in which 
vocational trainings were offered to the unemployed 
workforce, a second set of studies evaluated the 
effects of attending a vocational school. Compared to 
traditional schooling, Loyalka et al. (2013c) and Rozelle 
et al. (2015) found negative effects of vocational 
schooling in China, while Moenjak and Worswick 

(2003) found positive effects in Thailand, though the 
latter analysis is largely correlational (with selection 
bias corrections). More specifically, Loyalka et al. 
(2013c), using matching and instrumental variables 
analyses with a sample size of 10,000 students in China, 
found that attending vocational school (instead 

of a traditional high school) reduced general skills 

and had no effect on specific IT skills. They further 
found that low-ability and low-income students were 
more likely to drop out of vocational programs, as 
compared to academic programs. Similarly, Rozelle 

et al. (2015) found negative effects of vocational 
education in comparison to traditional high schools. 
Using the expansion of vocational education in China 
as a natural experiment, they employed a difference- 
in-difference approach to capture the returns of 
vocational schooling (versus regular schooling) on 
employment outcomes such as wages. They found that 
vocational education had inferior returns compared to 
general high schools. 


Further, two studies examined the impact of 

“winning” a voucher to attend a vocational school 
(versus losing the lottery) and found that while these 
programs seemed to have an impact on educational 
attainment, the impact on earnings was less clear (or 
unmeasured). First, Bettinger et al. (2010) conducted 
a randomized experiment in Colombia and found that 
among 5th grade students who applied to receive 

a vocational education voucher, those that won the 
lottery and thus had their pick of both private and 
public sector vocational schools were more likely to 
complete more years of schooling, less likely to repeat 
grades, 56 percentage points more likely to take the 
ICFES examination (university entrance examination), 
and more likely to have a higher reading score than 
students that lost the vocational education voucher 
lottery. The authors hypothesized that this was due to 
the fact that voucher winners were more likely to find a 
better match to their interest area given that they had 
their pick of all private sector vocational opportunities, 
whereas those that lost the lottery were more likely to 
attend public vocational schools or regular academic 
schools. The authors argued that voucher winners 
may have performed better due to the fact that the 


provision of the voucher (year to year) was dependent 
on student performance or because these private 
organizations may be better able to adapt their 
offerings to the demands of the labor market (in this 
case the growing service sector). 


In addition, Hicks, Kremer, Mbiti, and Miguel (2011) 
conducted a randomized experiment in which 
applicants to a technical and vocational vouchers 
program in Kenya were grouped into one control 
sample (half of the applicants) and two treatment 
samples (public school vouchers and private school 
vouchers, each representing one-fourth of the 

total students). They also randomly exposed half 

of treatment and control individuals to information 
about the actual returns to vocational education. 

In their preliminary analysis, the authors found that 
voucher winners were statistically more likely to enroll in 
vocational education and more likely to stay enrolled 
(with higher retention rates) than those awarded a 
restricted (public institution only) voucher. Further, 
among program completers, there is early, suggestive 
(but not statistically significant) evidence of earnings 
advantages for women but no overall impact on 
employment rates thus far. 


The only study that examined the impact of a 
vocational training on current employees found 
positive effects of these trainings on wage earnings. 
Specifically, Rosholm, Nielson, and Dabalen (2007) 
conducted a quasi-experimental evaluation (using 

a “matching” methodology) of on-the-job training 
within formal sector enterprises (a sample of 
manufacturing firms) in Kenya and Zambia. The authors 
found a positive rate of return to these trainings of 
approximately 20%; the effect was larger and more 
precisely measured for trainings done in larger firms 
and for trainings of longer duration. Further, the wage 
returns were larger for formal than for informal training 
in Kenya, while in Zambia, the reverse was true. 
However, there were some differences in background 
characteristics across control and treatment groups 
post-matching, thus the extent to which this estimated 
impact is purely causal comes into question. Also, the 
authors note the possibility that the training received 
by employees was concentrated among the most 
highly skilled, thus these results may not extrapolate to 
the workforce population as a whole. And finally, the 
authors estimated the effect of these trainings on those 
trained (a “treatment on the treated” effect), thus the 
average treatment effect would be expected to be 
smaller. 


Differential impact by gender. While the studies 
included here offer mixed findings overall, one pattern 
did emerge across all contexts, and that was that 
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women seemed to benefit more from these vocational 
education programs than men. While women may 
have been more reluctant to participate in vocational 
education programs (Hicks et al. 2011), their return on 
investment was often higher. Nine of the 15 studies in 
this worldwide synthesis explored gender differences 
in vocational education effects, and each found 
larger effects for females in the form of higher wages 
(Delajar, 2006; Chong & Galdo, 2006; Attanasio et 

al., 2011; Alzud & Brassiolo, 2006; Diaz & Jaramillo, 
2006), employment (lbarraran & Rosas-Shady, 2006; 
Attanasio et al., 2011; Aedo & Nunez, 2003; Alzud & 
Brassiolo, 2006; Delajara, 2006; Diaz & Jaramillo, 2006), 
formality of employment (Attanasio et al., 2011; Alzud, 
& Brassiolo, 2006), and income diversification (Chun & 
Watanabe 2011). 


Further, while women may have received larger 
returns from vocational education programs, they 
were less likely to sign up for a vocational education 
program, were more likely to sign up for more 
“female-dominated” vocational fields, and might not 
have been aware of the relative returns to various 
vocational sectors (Hicks et al., 2011). Examining their 
baseline data, Hicks et al. (2011) found that males 
have a strong preference for traditionally male- 
dominated fields (e.g., moter vehicle mechanics), 
whereas females tend to choose more female- 
dominated fields (e€.g., hairdressing). The authors stated 
that “given these apparent misperceptions about 
the returns to vocational training and the pervasive 
occupational segmentation by gender, the provision 
of additional information and encouragement 

could have meaningful consequences for individual 
educational investment choices” (p. 4). 


Thus the authors included in their vocational 
education information session a portion devoted to 
highlighting the disparity between the average earning 
of graduates in male-dominated versus female- 
dominated trades. They also used a video featuring a 
successful female car mechanic in order to encourage 
women to choose these more highly-rewarded 

trades. The authors found that females exposed to the 
information intervention were almost nine percentage 
points more likely to express a preference for a male- 
dominated course, and five percentage points more 
likely to actually enroll in one; younger and more 
educated females were especially likely to prefer 
male-dominated fields. Thus to the extent that women 
are better informed about the relative returns to 
various types of vocational programs and are strongly 
encouraged to participate, there is the potential for 
strong growth in these vocational fields. 
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Public versus private vocational programs. There is 
limited and only suggestive evidence that private 
vocational programs may yield better results than 
public programs. As described above, Bettinger et 

al. (2010) found that 5th grade voucher winners had 
higher completion rates, higher retention rates, a 
higher probability of taking the university entrance 
examination, and higher reading achievement than 
students that lost the vocational education voucher 
lottery. Again, the authors hypothesized that this is due 
to the fact that voucher winners were more likely to 
find a better match to their interest area given that 
they had their pick of all private sector vocational 
opportunities and that these private organizations may 
be better able to adapt their offerings to the demands 
of the labor market (in this case the growing service 
sector). Also in Colombia, Medina and Nunez (2005), 
using a propensity score matching methodology, 
evaluated the impact of public versus private 
vocational programs on student outcomes and found 
that private institutions offer a larger wage advantage 
to adults (both genders), as well as to male youth. In 
addition, Hirshliefer, McKenzie, Almeida, & Ridao-Cano 
(2014), in a randomized trial of vocational programs for 
unemployed youth in Turkey, found that the short-term 
impact of the training on the quality of employment 
was positive and statistically significant, and that 

this impact was even stronger when the training was 
offered through private providers. However, of note are 
that these gains were not sustained past three years, 
and that the program had only a slightly positive and 
statistically insignificant impact on employment itself. 


Hicks et al. (2011) conducted a similarly designed 
experiment in Kenya, but only their preliminary results 
are available. Thus far, they found that individuals who 
were awarded restricted vouchers (these could be 
used for public institutions only) were approximately 
16 percentage points more likely to drop out, 
compared to those who were awarded unrestricted 
vouchers (these could be used for either public or 
private institutions). The authors argued that this 
made sense, given that restricted vouchers are less 
flexible to individual choice but that it is too early 

to make conclusions about differential employment 
across treatment groups (the program is still in the 
implementation phase). 
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Limitations and Implications 


Drawing from the above sample of impact evaluations, 
it seems clear that vocational education programs 
benefit from tight linkages to targeted industries or 
firms—either through an internship or on-the-job portion 
of a training program (preferably paid so that the 
industry has an incentive to hire the student in which 
they have already invested); the tighter this link, the 
more successful the intervention from the standpoint 

of the student (Card et al., 2011; Chun & Watanabe, 
2011). Ideally, industries would use this portion of 

a vocational training program to screen for future 
employees, or vocational education programs could 
work as a “feeder” program into firms. 


Because high quality training institutions have been 
shown to have a much higher impact on students, 

it is recommended that vocational programs 

included in any large-scale intervention be chosen 
through a competitive bidding process in which the 
following factors may jointly determine program 
quality: legal status (formality), the level of human 
resources and infrastructure, physical equipment, 
appropriate coursework or curricular structure, 

class size, expenditures per trainee, teacher quality 
variables (experience, education, etc.), and variables 
characterizing the link between the content of the 
training courses and the institution’s knowledge about 
workers and occupational analysis of labor demand 
(Attanasio et al., 2011; Chong & Galdo, 2006; Diaz 

& Jaramillo, 2006). In addition, if the organization in 
charge of implementing the vocational education 
program can also function as a job placement and 
counseling organization, this can be an efficient way 
in which to track students and ensure their success. 
Furthermore, policymakers should not rule out the 
possibility of working in some capacity with the private 
vocational sector as there is preliminary or limited 
evidence that private vocational schools may be more 
able to adjust to changing market demands. That said, 
in the case of an economic stagnation, it is possible 
that individuals that trained in private vocational 
schools may face structural unemployment due to the 
more targeted (and thus narrower) skill set offered by 
these demand-driven institutions. 


Overall, it seems that a country’s economic context 
(for example, level of unemployment, market 
structure, or level of informality or formality) may 
determine the level of success of various vocational 
programs, possibly more so that then characteristics 
of these interventions themselves (this may explain 
the very mixed results from the 35 different vocational 
education evaluations). However, regarding gender, 


certain patterns do emerge: women’s presence in the 
vocational education field seems to be an untapped 
resource as women consistently seem to benefit more 
from vocational training than their male counterparts 
(Aedo & Nunez, 2003; Alzud & Brassiolo, 2006; Attanasio 
et al., 2011; Chong & Galdo, 2006; Chun & Watanabe 
2011; Delajar, 2006; Diaz & Jaramillo, 2006; Ibarraran & 
Rosas-Shady, 2006) but are also less likely to sign up for 
vocational training or, if they do sign up, are less likely 
to choose the vocations with higher returns (currently 
these are male-dominated fields) (Hicks et al., 2011). 
Recommendations thus include heavy marketing to 
women to increase their participation in the vocational 
sector and the need to encourage women to take up 
vocations with higher rates of return. 
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Apprenticeships 
Katharine Conn 


An apprenticeship is a form of on-the-job training 
that is characterized by training under an employer 
or master craftsperson in exchange for a period 

of continuous labor. Apprentices often gain their 
license or accreditation to practice their trade 

or profession upon completion of their training. In 
some cases, on-the-job labor is accompanied by 
more formalized coursework or study. Formalized 
apprenticeship programs are very common outside 
of the Unites States where vocational education is 
the more dominant form of industry-specific training. 
Consequently, the majority of the rigorous literature 
in this area comes from Europe in particular. This 
section explores the efficacy of apprenticeship 
training programs. It includes a description of the 
literature as a whole (program targets, publication 
types, outcomes, methodologies, regions covered, 
etc.), the relevant findings associated with this body 
of work, and any limitations or implications of these 
studies. Note that while informal apprenticeship 
programs are common in developing countries 
(particularly Africa)*®, there is limited experimental or 
quasi-experimental literature on the success of this 
type of informal training. 


Literature Description 


Twenty-seven efficacy studies regarding the impact of 
apprenticeships on youth outcomes were identified for 
inclusion in this synthesis.” In addition, one systematic 
review examining the impact of apprenticeships 

on life outcomes in Europe was also included in this 
review. The following section provides an overview of 
the characteristics of the studies found through the 
systematic review process. 


Study Population. Of the 27 high-quality apprenticeship 
studies identified for this synthesis, all 27 targeted 
apprentices or trainees in the workforce. 


Publication Type. Of the 27 high-quality studies 
identified under the apprenticeship approach, 
18 are journal articles, while the remaining 9 are 
working papers. 


6 Fora discussion of these informal apprenticeships see Filmer, D., & Fox, L. (2014). Youth Employment in Sub-Saharan Africa 


The World Bank: Washington D.C. 


7 See Appendix C for a description of the search strategy and search terms. 
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Outcomes. Most of the outcomes examined in the 
apprenticeship literature focus upon individual- 

level wages, wage profiles, length of time ata 

job, successful transition to work/ employment 
status, unemployment spells, participation in further 
education, responsiveness to firm closure, and 
promotion to management. Macroeconomic 
outcomes explored in these studies include country- 
level job turnover, employment growth, employment 
level, effects on capital, investment and skill ratio, and 
productivity (value added per worker and sales per 
worker). 


Methodology. Of these studies, 8 used causal quasi- 
experimental methods such as matching, regression 
discontinuity, or instrumental variables, while the others 
employed a large range of rigorous non-experimental 
methods such as dynamic discrete choice models, 
survival analysis, multivariate probit models controlling 
for endogeneity and self-selection, or proportional 
hazard models. 


Region. Of these 27 apprenticeship studies, 14 take 
place in Germany, 3 in Switzerland, 2 in Austria, 1 joint 
study in both Germany and Switzerland, and 1 study in 
each of the following countries: France, Denmark, the 
Netherlands, Italy, Romania, the United States, and the 
United Kingdom. 


Findings 


It is important to note that the efficacy literature on 
apprenticeship training come from the developed 
world, and with the exception of one study in the 
United States, this literature is from Europe (Germany 
in particular). This is due to the fact that formal 
apprenticeship programs are well established in this 
region of the world and supported by the European 
Centre for the Development of Vocational Training 
(Cedefop), and while they may be spreading to other 
regions (Singapore for example), rigorous evaluations 
of these programs are essentially limited to Europe. 


The findings of these studies cover seven major themes: 
country-level factors influencing apprenticeship 


provision, the positive rates of return to apprenticeships, 


the impact of apprenticeships on transition to 
employment and length of employment, whether there 
is areal apprenticeship advantage, the long-term 
benefits of apprenticeships, the heterogeneous effects 
associated with apprenticeships, and the impact of 
apprenticeships on the macro economy. 


Couniry-level factors influencing apprenticeship 
provision. As to why apprenticeship programs are 
so widespread in only certain countries, Acemoglu 
and Pischke (1998) argued that whether or not firms 
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provide apprenticeship training is related to general 
equilibrium variables such as job turnover (as well as 
the cost of firing workers) within a county. For example, 
in the United States where job turnover is high, this may 
be a disincentive for a high training outcome, as firms 
will not be willing to invest in workers that have a high 
probability of leaving, whereas in Germany where 
workers do not quit as often, there is a higher incentive 
to offer apprenticeship programs. Interestingly, the 
authors noted that while the investment margin is less 
distorted in a high-training equilibrium, this scenario 
may still result in a worse allocation of worker-job 
matches because workers may not be free to pursue 
their interests and needs. 


Other factors influencing the scope of apprenticeship 
offerings include economic downturns; Brunello (2009) 
found that countries/firms are less likely to increase 
their share of new apprentices during a recession; 
they instead shifted their focus to training incumbent 
staff. Further, the costs associated with apprenticeship 
programs also depend on country-level factors such 
as relative wages between types of workers and the 
share of productive tasks given to apprentices within 
firms; these main factors are said to account for the 
differential costs of German versus Swiss firms offering 
apprenticeship programs (Wolter, MUhlemann, & 
Schweri, 2006; Dionisius et al., 2009). 


Positive rates of return. Numerous studies reported 

the positive upfront effects of apprenticeships on 
wage returns. Three studies from West Germany and 
Germany, spanning three decades reported positive 
rates of return to apprenticeships: Adda, Dustmann, 
Meghir, and Robin (2010) for West Germany from 
1975-1996; Clark and Fahr (2002) for Germany from 
1984-1995; and Werwatz (2002) for Germany in 1996. In 
addition, McIntosh (2007) measured positive rates of 
return to apprenticeships in the United Kingdom from 
1975-1998; Rezin and McCaslin (2001) found positive 
wage returns to apprenticeship programs in the United 
States (Ohio) from 1993-1998; and Fersterer, Pischke, 
and Winter-Ebmer (2008) found significant earnings 
returns in Austria for the period 1975-1998. 


Impact on transition to employment and length of 
unemployment. Two studies cited the impact of 
apprenticeships on finding immediate employment: 
Bonnal, Mendes, and Sofer (2002) in France and Rezin 
and McCaslin (2001) in the United States (Ohio), while 
Malamud and Pop-Eleches (2010) found that there is 
no advantage for apprenticeship programs (versus 
general education) in Romania from 1995-2000. 
Regarding the length of unemployment, Bonnal et al. 
(2002) (France), Hofer and Lietz (2004) (Austria), and 
Parey (2009) (Germany), found that apprenticeships 


tend to reduce the amount of time spent between 
employment spells. 


Is there a real apprenticeship advantage? When the 
returns to apprenticeship programs are compared 
against the returns to a high school diploma or a more 
traditional vocational education program, it is unclear 
if there is an apprenticeship advantage, though there 
is only limited evidence in this area. Parey (2009) found 
that there is no significant difference in wage returns 
between vocational school graduates and apprentices 
in Germany between 1975-2001. And in Austria, Hofer 
and Lietz (2004) found an apprenticeship advantage 
over individuals who complete only compulsory 
education; however this advantage disappears (and 
becomes negative) when comparing apprenticeship 
returns to those of high school graduates (from 1993- 
1998). Finally, in Switzerland, Bertschy, Cattaneo, and 
Wolter, (2009) found that full-time trainees in vocational 
schools are in fact less likely to find an appropriate job 
match than those who completed apprenticeships. 


Benefits of apprenticeships may diminish over time. The 
long-term benefits of apprenticeships programs are 
not as strong, on average, as the short-term benefits. 
Adda et al. (2010) found that apprenticeships in West 
Germany increase wages and alter wage profiles with 
more growth initially, while wages for non-apprentices 
grow at a lower rate but for longer. In France, Bonnal 
et al. (2002) also found that while apprenticeships have 
a positive impact on employment duration overall, 

for those individuals who do not find employment 
immediately, having completed an apprenticeship 
becomes a “penalty rather than an advantage” 
(though this applies to only a small subset of the 
apprenticed students) (p. 438). Further, in Germany, 
Caliendo, KUnn, and Schmid! (2011) found that after 
30 60 months, apprenticeships have a negative effect 
on employment (though a positive impact on further 
education). Hofer and Lietz (2004) in Austria found that 
the differential between wage returns of high school 
graduates and former apprentices tends to widen over 
time (in favor of high school graduates). And finally, 
Groot and Plug (1998) examined the long-term impact 
of apprenticeships versus school based vocational 
education in the Netherlands, and found that there 
are no significant differences between these groups in 
terms of earnings, earnings growth, or employment for 
years 1985-1994. 
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Heterogeneous effects of apprenticeships. The 
impact of an apprenticeship program on wages 

or employment may also vary by factors such as 
previous educational performance, occupational 
choice, gender, or size of training firm. Regarding 
previous performance, Buchel (2002) found that 
students with a high-quality school-leaving certificate 
in West Germany were more likely to have access to 
a “good” apprenticeship and were also less likely to 
experience negative effects regarding the school- 
to-work transition. Further, Goggel and Zwick (2012) 
found that within industrial occupations, individuals 
who change employer or occupation enjoy wage 
advantages, while within the fields of commerce, 
trade, construction, and crafts, individuals face large 
wage losses from an occupational change. 


Regarding gender differences, Hofer and Lietz (2004) 
found that the wage returns to female apprentices 
are not significantly different from those of female 
unskilled workers in Austria; however, they did find that 
apprenticeship training does seem to assist women 

in finding more stable occupations. The authors 
argued that this finding stems from the fact that 
women who complete apprenticeships do so largely 
in more traditional low-return fields such as the service 
sector. Bonnal et al. (2002) found that the impact of 
apprenticeship trainings on employment probability is 
higher for men than for women. The authors were not 
able to account for this differential but do say that for 
women, having been trained in business or the hotel 
trade, facilitates employment; thus it is possible that 
women either do not pick apprenticeships in other 
fields, that women are more likely to search for part- 
time work at a lower pay, or that women have difficulty 
getting hired in non-traditional fields. 


Finally, regarding heterogeneous effects on stayers 
(those who remain at their training firm post- 
apprenticeship training) versus movers (those who 
leave), Acemoglu and Pischke (1998) in Germany (for 
years 1979-1985) found that stayers earned higher 
wages than those who leave their apprenticeship firm 
for other reasons (though not a statistically significant 
result at the 5% level). Conversely, Euwals and 
Winkelmann (2004) also in Germany, argued that while 
stayers have longer first job durations than movers, 
this advantage actually does not hold for wages 
during the years 1975-1995. The authors also found 
that the intensity of the apprenticeship (as proxied 

by wage rate and training length) was positively 
correlated with retention rates, first job durations, and 
post-apprenticeship wages. Finally, Bougheas and 
Georgellis (2004) in West Germany, found that among 
occupational movers alone, apprentices trained in 


Consortium for Policy Research in Education | cpre.org 


Strategies for Strengthening the Technical Workforce 


large firms earned a higher wage than those in small 
firms. 


Effects of apprenticeships on the macro economy. In 
addition to the effects of apprenticeship programs 
on individuals, these initiatives also have an impact 
on overall labor market statistics. Only one study was 
able to study such macro effects: In Italy, Cappellari, 
Dell’Aringa, and Leonardi (2012) studied the effects 
of the 2001 reform which liberalized apprenticeship 
contracts. This reform increased the turnover and the 
net employment of apprentice workers and reduced 
the cost of apprenticeship contracts so that firms 
were encouraged to substitute external temporary 
staff with apprentices. They found that this reform 
has had an overall positive effect on the economy 
with an increase in employment growth (+1.6%), 

job turnover (+3%), employment level (+5%), value 
added per worker (+1.5%), sales per worker (+0.9%), 
and total factor productivity (+1.6%). The effects on 
capital, investment, and skill ratio were not statistically 
significant however. In addition, Origo and Patrizio 
(2013) conducted a cross-country analysis using 
European panel data to explore the relationship 
between apprenticeship indicators and youth labor 
market outcomes and found that a higher frequency 
of apprenticeships was associated with higher youth 
(15-24) employment rates. 


Limitations and Implications 


As the rigorous efficacy literature on apprenticeship 
programs comes only from Europe, where these 
programs have a multi-decade history, drawing strong 
inferences from these programs to the rest of the world, 
including the developing world, is not recommended. 
However, within the European legal, political, and 
economic context, the following points synthesize the 
bulk of the literature addressed above: 


e The decision to initiate national apprenticeship 
programs (and for firms to initiate apprenticeship 
programs) is based on numerous country-level 
factors such as job turnover rates (Acemoglu & 
Pischke, 1998), trends in economic growth (Brunello, 
2009), and relative wages (Wolter, MUhlemann, & 
Schweri, 2006; Dionisius et al., 2009). 


e Apprenticeships are linked to positive income 
returns, particularly in the short term (Adda et 
al., 2010; Clark & Fahr, 2002; Fersterer et al., 2008; 
Mcintosh, 2007; Rezin & McCaslin, 2001; Werwaiz, 
2002), though in the long term these returns may 
decline (Adda et al., 2010; Groot & Plug, 1998; Hofer 
& Lietz, 2004). 


e Having completed an apprenticeship is associated 
with shorter unemployment spells (compared 
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to unskilled workers) (Bonnal et al., 2002; Hofer 

& Lietz, 2004; Parey, 2009) and an increased 
probability of finding immediate employment 
(post-apprenticeship) (Bonnal et al., 2002; Rezin & 
McCaslin, 2001). 


e When compared to individuals who have 
completed secondary education or vocational 
education training (VET), the results for 
apprenticeship training are more mixed (Parey, 
2009; Hofer & Lietz, 2004; Bertschy et al., 2009). 


e Returns to apprenticeship training are higher 
for men (partly due to the fact that women are 
more likely to complete apprenticeships in more 
traditional and lower-wage sectors) (Bonnal et 
al., 2002; Hofer & Lietz, 2004), and there is limited 
evidence that apprenticeship returns are higher 
for students with initially high quality education 
credentials and for those that complete more 
“intense” or prestigious apprenticeships. 


e There are very limited findings regarding the 
macroeconomic impact of apprenticeship reforms, 
but evidence from Italy suggests that liberalizing the 
apprenticeship system may generate productivity 
returns (Cappellari et al., 2012). 


Finally, a new randomized study, whose 
implementation is currently underway, attempts to 
measure the impact of a newly established national 
apprenticeship program in Ghana (Lehrer, Mbiti, 

& McCasland, 2014). This will be the first national 
apprenticeship program with a rigorous evaluation 
outside of Europe. An innovative payment scheme 
accompanies this nation-wide program in which 
providers (firms) are paid based on their performance 
(based on the skill level and employment outcomes 
of their apprentices). The cost-effectiveness of this 
program will also be evaluated by the authors in 
order to inform other developing countries about the 
potential pay-offs associated with such an endeavor. 


Cooperative Education 
Elizabeth Park 


This section explores the efficacy of cooperative 
education, a form of work-based learning which 
integrates classroom instruction with alternating 
periods of paid discipline-related work experience. 
Cooperative education programs are normally 
offered at the post-secondary level. In 1906, during 
his tenure as Dean of the University of Cincinnati, 
Herman Schneider first developed the concept of 
cooperative (“coop”) education with the first coop 
class of 27 engineering students and 13 companies 
(Cooperative Education & Internship Association, 
n.d.). Coop education has since grown to be widely 
accepted in the United States and beyond in fields 
other than engineering (e.g., business, health, 
liberal arts). Coop education structures academic 
education with practical work experience by 
providing academic credit for work experience. 
Coop students typically participate in school-work 
rotations that may span as many as three academic 
years in four-year colleges (shorter periods in 
community colleges and high schools). The coop 
experiences are either full-time (40 hours of work per 
week) with alternating periods (semester, quarter) 
of work and school, or part-time (20 hours per week) 
combining work and school during the same time 
period. The underlying idea is that by working in 
professional environments during their student years, 
students will be more committed to their career 
choice, be more motivated to study, and acquire 
strong work transition skills. 


Literature Description 


Seventeen studies on the impact of coop education 
were identified for inclusion in this review. The following 
section provides an overview of the characteristics of 
these studies. 


Study population. Three of the 17 studies looked at high 
school students or high school graduates. The majority 
of the remaining studies looked at students at the 
post-secondary level including university, college, and 
community college students. 


Publication Type. Of the 17 coop education studies 
identified for this synthesis, 16 are journal articles. One is 
a synthesis and advocacy report. 

Outcomes. Most of the studies focused on the 
students’ performance during schooling (GPA, 
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duration of completing education, retention), returns 
to cooperative education (wages, condition of 
employment status), and on students’ psychological 
conditions, including perception of the program and 
feelings of self-efficacy. 


Methodology. None of these studies employed 
randomized controlled trials, and only one employed 
some form of matching (propensity score matching). 
The remaining studies employed a mixture of methods 
including comparisons of non-equivalent groups and 
regression analysis with controls. Data were collected 
through school records and/or student surveys. 
Region. The studies in this sample are mainly from the 
United States, with the exception of 3 studies from 
Canada. 


Findings 


This section synthesizes the findings of these coop 
education studies into the following major themes: 
coop education at the secondary school level, coop 
education at the post-secondary level, completion 

of the program and retention at school, work-related 
outcomes, impact on job placement and persistence, 
and impact on self-efficacy. 


Cooperative education at the secondary school level. 
Coop education programs at the secondary school 
level in the United States expanded when the federal 
Government passed the School-to-Work Opportunities 
Act (STWOA) in 1994. This act provides initiatives for 
work-based learning activities and opportunities 

that bridge the gap between school and work. This 
act also provides funds to states to establish coop 
education programs and provides financial incentives 
to employers to hire coop education students (Richard, 
Walter, & Yonder, 2013). 


Several studies have explored the impact of 
cooperative education on a variety of outcomes. 
One research synthesis of studies completed after 
1994, focused on research reporting on the impact 

of the school-to-work initiative on students, teachers, 
and employers. The synthesis suggested that, overall, 
students who participated in work-based learning had 
greater attendance, were less likely to drop out of 
school, and were more likely to view work as a way to 
learn new things and prepare for the future (Hughes, 
Bailey, & Mechur, 2001). Further, an analysis of the 
impact of a sixty-hour work-based internship program 
at a large Midwest urban school district in the United 
States found that social support from adult supervisors, 
and in particular student mentors, positively influenced 
students’ occupational engagement (Bennet, 2007) 
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(Note that this analysis was based on self-reported 
student survey data alone). 


Another study using propensity score matching 
(between coop education participants and non-coop 
education participants) and data from the Educational 
Longitudinal Study of 2002 found that participating in 
cooperative education in high school had a significant 
positive effect on at-risk students’ post-secondary 
education plans including students’ plans to go 

to school right after high school, plans to continue 
education at some point in the future, and the type 

of school they planned to attend (Gemici & Rojewski, 
2010). However, they found no difference in students’ 
thoughts regarding the importance of work, including 
the importance of gaining steady employment, 
obtaining a good job, being successful in one’s chosen 
line of work, and being an expert in one’s field by 
participation in a cooperative education program. 


Findings regarding the impact of coop education on 
high school students’ academic performance have 
been mixed. Exploring whether coop and non-coop 
students’ GPA varied based on the number of hours 
worked per week, Stern, Finkelstein, Urquiola, and 
Cagampang (1997) found that among high school 
students, working longer hours was associated with 
lower academic performance, but the negative effect 
was much less pronounced among coop students. A 
study of career and technical education examined 
the extent to which high school students’ participation 
in cooperative education is correlated with their 
scores On National Occupational Competency 

Testing Institute (NOCTI) exams—a measure of the 
effectiveness of career and technical schools (Richard 
et al., 2013). Students who participated in a coop 
program scored significantly higher on both written 
and practical portions of the NOCTI overall, as well as 
within individual program types with the exception of 
the carpentry and the childcare programs. 


Cooperative education at post-secondary level. 
Several studies investigated the association between 
participating in a cooperative education program 

at the post-secondary level and school outcomes, 
including GPA and duration in school. Blair and Millea 
(2004a) found that among engineering and business 
students at Mississippi State University (MSU), coop 
students had higher average GPAs than non-coop 
students, though this finding did not hold true for other 
majors. However, note that this study did not compare 
student populations that were equivalent; coop 
students were compared to students who may or may 
not have been eligible for the coop program. Blair 
and Millea (2004b) also reported that students who 
successfully completed the cooperative education 
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program (defined as completing all three semesters) 
demonstrated higher academic performance as 
measured by their GPAs than students who were not 
enrolled in cooperative education. However, they 
found no difference between the GPAs of students who 
did not complete the coop program and those who 
did not enroll in the program at all (again this study 
also did not compare equivalent student populations). 
Looking specifically at MSU engineering majors, Blair, 
Millea, and Hammer (2004) found that coop engineers 
earned higher average GPAs than their non-coop 
engineering peers. However, to account for the 
potential bias of academic performance requirements 
for cooperative education programs, they compared 
coop eligible students with coop enrolled students, 
and found that students who successfully complete the 
cooperative education program graduate with higher 
GPAs than their non-coop peers who were eligible to 
enroll in the program. Although the results of these 
studies are somewhat mixed, overall they find that 
completion of a coop program at the post-secondary 
level has a positive impact on academic performance. 


Completion of the program and retention at 

school. Studies examining the relationship between 
cooperative education and duration in school found 
that participation in coop programs extended the 
time needed to graduate by about two to three 
semesters (Blair & Millea, 2004b; Blair et al., 2004). 
However, over half of that time was spent gaining work 
experience through the program. It was hypothesized 
that compensation from the work periods in coop 
education programs could help pay for some part 

of students’ educational expenses. Thus, it could be 
speculated that students might need to stay longer to 
complete the degree. This topic of inquiry, however, 
was not identified through the systematic literature 
search. 


Avenoso and Totoro (1994) looked at the impact 

of coop education on students’ retention in school 

by examining data from four entering classes from 
1989—1992 at Long Island University’s Southampton 
Campus. They compared two groups—students who 
were placed in cooperative education programs as 
freshmen and sophomores versus their non-coop peers. 
Based on the official college records, the authors 
found that the retention rate of the cooperative 
education students was significantly higher than that of 
the non-coop students. Although the authors checked 
the equivalency of the two comparison groups at 
baseline by comparing their SAT verbal scores, this 

is not enough to establish that the two groups were 
comparable across all observable characteristics. Thus, 
the issue of selection bias, coupled with the study’s 
small sample size warrants caution in assuming a causal 


relationship between retention and the cooperative 
education program. Finally, Jaeger, Eagan, and Wirt 
(2008) examined the influence of coop education on 
the persistence of science, math, and engineering 
students entering in 1997 and 1998 at a large public 
university. The study found that 27% of all students 
dropped out before completing the degree. Students 
who participated in a coop education program were 
more than five times as likely to be retained compared 
to their peers who did not have coop education 
experience. 


Work-related outcomes. Earlier non-experimental 
studies on the impact of cooperative education 

on starting salaries reported mixed results. Dubick, 
McNerney, and Potts’ (1996) analysis of computer 
science majors from a Midwest university found that 
graduates with cooperative education experience 
had higher starting and current salaries than their peers 
without Coop experience. In contrast, accounting 

for pre-existing differences between coop and non- 
coop students among both high school and two-year 
college students, Stern et al. (1997) found no difference 
in earnings. Using data form North Carolina community 
colleges, Wessels and Pumphrey (1996) examined the 
impact of the cooperative education experience, as 
well as the institutional effect of attending a school 
that offers cooperative education, on starting salaries; 
the institutional effect of coop programs increased 
wages by 7% for females (p < .05), while there was 

no direct effect from participating in cooperative 
education itself. For males there was no difference 

in the starting salaries between coop and non-coop 
students, or between students attending a school 
offering coop programs and those attending a school 
that did not offer coop programs. Further, Blair and 
Millea (2004a) compared 5,506 coop and non-coop 
engineering, business, and other program graduates 
at the Mississippi State University and found that on 
average, coop students earned significantly more 
(over $6,000) than non-coop students. However, the 
salary benefits associated with cooperative education 
varied across majors. In engineering for example, 
coop graduates on average earned more than their 
non-coop peers, but business majors in cooperative 
education programs earned less than their non-coop 
peers. 


Three non-experimental studies explored earnings 
among Canadian college and community college 
students. An earlier study by Darch (1995) analyzed 
data from a survey of 1990 graduates and found that 
among those who were employed at the time of the 
survey, math and physical science graduates earned 
$5,490 CDN more than non-coop grads in the same 
field. For commerce and economics, there was a 
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$3,700 CDN premium for attending a coop program. 
Studies drawing on more recent data also found 
graduates of coop programs earned significantly 
more than their counterparts in traditional non-coop 
programs (Goho & Rew, 2009; Walters & Zarifa, 2008). 
Drawing on the National Graduate Survey from 
Statistics Canada in 2000, Walters and Zaria (2008) 
found that the earnings advantage for graduates with 
cooperative credentials was particularly strong among 
university graduates; they earned approximately 
$8,000 CDN more per year than their counterparts with 
traditional (non-coop) undergraduate credentials. 
Similarly, Goho and Rew (2009) examined data from 
Canadian community college students and their 
analysis indicated that recent cooperative program 
graduates were more likely to be employed, to be 
employed in positions related to their education, and 
to have somewhat higher earnings than non-coop 
program graduates. Graduating from a cooperative 
education program resulted in an increase in monthly 
wages of about 4.8%. 


Impact on job placement and employment retention. 
Studies find that coop programs have a positive 
association with job placement and employment 
retention (Dubick et al., 1996; Wessels & Pumphrey, 
1995). Dubick et al. (1996) found that computer 
science graduates who participated in a coop 
program found employment more quickly than 

their non-coop peers; Coop graduates were able to 
find employment in nearly two months less. While it 
took four months for coop graduates to reach 90% 
employment, it took non-coop graduates an average 
of nine months. Interestingly, Wessels and Pumphrey 
(1995) found that for North Carolina community 
college students, graduating from a college that 
offered coop programs was more of a factor in 
decreasing job search time than actually participating 
in a coop program. Furthermore, they also found that 
approximately 40% of coop students found their first job 
with their coop employer. 


While the speed of job placement is important, job 
retention is equally important. Dubick et al. (1996) 
examined the relationship between participation in 
cooperative education and job turnover. Their results 
suggest that non-coop graduates changed employers 
significantly more often than cooperative education 
graduates. They also report that the combined effect 
of attending a college with a coop program and 
enrolling in a coop program was significant and 
positive for job advancement. However, according to 
Wessels and Pumphrey (1995), cooperative education 
has little impact on turnover. 
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Impact on self-efficacy. Raelin et al. (2011) collected 
survey data on self-efficacy, which is defined as 

an individual's perceived level of competence to 
complete a task. In particular, this study formulated 
self-efficacy as a composite of work self-efficacy 
(confidence within the workplace), career self- 
efficacy (confidence within their chosen engineering 
career), and academic self-efficacy. Data were 
drawn from sophomore engineering students in four 
universities: Northeastern University, Rochester Institute 
of Technology, Virginia Polytechnic Institute, and the 
University of Wyoming. The authors found that overall 
self-reported ratings of work self-efficacy and career 
self-efficacy (though not academic self-efficacy) 
increased during students’ sophomore years, regardless 
of whether they participated in a coop program or not 
(p < .01). However, they also found that completing a 
cooperative education program had a positive impact 
on the work self-efficacy of students (p < .01), whereas 
work self-efficacy decreased somewhat for those who 
did not participate in a coop program. They found 

no relationship between participation/completion in 
cooperative education and career or academic self- 
efficacy (p > .05). 


Limitations and Implications 


As our systematic search yielded few studies using 
rigorous methods, it is difficult to reach a research 
consensus regarding the effectiveness of such 
programs. With such a small number of experimental 
or quasi-experimental studies, the results reported here 
may not be purely causal due to selection bias; that 
is, the individuals who choose to attend a cooperative 
education program may not be comparable to those 
that don't, thus any differences between these groups 
may not be differences fully attributable to the coop 
program alone. Further, the data used in a number of 
these studies was self-reported, which is potentially less 
reliable. 


From the limited evidence available, however, it seems 
that cooperative education is a model that deserves 
consideration, as it appears to have high potential for 
connecting school experience with the workplace and 
for keeping students in schools. A number of the studies 
examined here indicated that students who completed 
cooperative education received higher starting salaries 
compared to non-cooperative education students 
(Blair & Millea, 2004a, 2004b; Blair et al., 2004; Darch, 
1995; Goho & Rew, 2009; Walters & Zarifa, 2008) and 
that coop education graduates spent less time finding 
a first job placement due to their prior exposure to 

the industry (Dubick et al., 1996; Wessels & Pumphrey, 
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1995). Many cooperative education graduates found 
that the program raised their confidence level at 
work and helped them stay in school (Raelin et al., 
2011). Additionally, cooperative education students’ 
responses to program surveys indicated several factors 
that might make cooperative education programs 
stronger, including the importance of offering detailed 
guidance about industries in which students could be 
placed for work, providing continuous mentoring by 
adults while working, and implementing high-quality 
transition programs (Garavan & Murphy, 2001; Nasr, 
Pennington, & Andres, 2004). 


Career Academies 
Wakasa Nagakura 


This section explores the efficacy of career 
academies, which are popular in the United States 
and have spread to several other countries. Career 
academies offer a form of work-based learning 
within an academic secondary school. Typically, 
these programs have career themes, show students 
the links between their academic subjects and a 
potential career, and involve employers and higher 
education institutions in preparing students for 
college and a career. A career academy usually 
operates in grades 9-12 or 10-12, and includes 
required academic subjects as well as career 
related classes, in order to ensure that students 

are eligible to enter university. Career academies 
encourage students to go to both 2- and 4-year 
colleges. In short, the three fundamental features of 
career academies are small learning communities, 
a college-preparatory curriculum with a career 
theme, and partnerships with employers and post- 
secondary educational institutions (Stern, Dayton, 
& Raby, 2010). Their main objectives are to prevent 
students from dropping out and to encourage 
students to advance to post-secondary education 
or the workplace. This section includes a description 
of the career academy evaluation literature as a 
whole, the relevant findings associated with this 
body of work, and any limitations or implications of 
these studies. 
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Table 1: MDRC Studies/Reports on Career Academies 


Year Study title/Author Summary of findings 
1997 Career Academies: Communities of 1. The Career Academies (CA) provided their students and teachers with 
support for students and teachers— a greater degree of interpersonal support and a strong professional 
Emerging findings from a 10-site community. 
evaluation (Kemple) 2. CAstudents were more likely to combine academic and career 
goals by participating in career awareness and work-based learning 
activities. 
2000 Career Academies: Impacts 1. CAs enhanced the degree of interpersonal support for students from 


on Students’ Engagement and teachers and peers. 
Performance in High School (Kemple 2. Overall, more than a third of the CA students left the program before 
& Snipes) the end of their 12th grade year. 
3. Effectiveness varied among different risk level groups: the CA group had 
a lower dropout rate (21%) than the non-CA group (32%) and a higher 
average attendance rate (82%) than the non-CA group (76%). 
4. CAs did not improve students’ standardized test scores (reading/math). 


2001 Career Academies: Impacts on 1. Relative to similar students nationally, both CA and non-CA groups 
Students’ Initial Transitions to achieved higher rates of high school graduation, college enrollment, 
Post-secondary Education and and employment. 

Employment (Kemple) 2. The two groups were comparable with respect to dropout rates from 
post-secondary education. 


2004 Career Academies: Impacts on Labor 1. The CAs substantially improved the labor market prospects of young 
Market Outcomes and Educational men. The young men in the CA group earned over $10,000 (18%) more 
Attainment (Kemple) than those in the control group over the four-year follow-up period. 
2. The CAs had no significant impact (positive or negative) on the labor 
market outcomes for young women. 


2008 Career Academies: Long Term 1. CA graduates earned 11% more than non-CA members on average 
Impacts on Labor Market Outcomes, ($2,088 a year; $16,704 over 8 years). 
Educational Attainment & Transitions 2. The impact was particularly strong for men, whose earnings increased 
to Adulthood (Kemple) by $3,371 (17%) per year or nearly $30,000 over 8 years. 
3. CAstudents demonstrated higher attendance, increased academic 
course taking and better on-time graduation rates. 


Methodology. The methodologies ranged from 
Randomized Control Trials (RCTs) to matching 
group designs, and employed both quantitative 
(administrative student data), as well as qualitative 
data (interviews and classroom observations). 


A. Literature 


Seventeen high-quality studies were included in this 
synthesis. The following section provides an overview 
of the characteristics of the studies found through the 


systematic review process. Outcomes. These studies assessed a wide variety of 


student outcomes; these include both academic 


Study Population. The study population for this set of 
studies consisted of high school students (9th-12th 
grade); multiple studies tracked these students through 
their post-secondary institutions and into the labor 
market as well. 


Publication Type. Six of the studies in this section are 
journal articles; the remaining studies are working 
papers and evaluation reports. 


outcomes (high school graduation rates, dropout rates, 
attendance, academic achievement, enrollment 

in post-secondary institutions, and increased-course 
taking) as well as labor market outcomes (employment 
status and earnings). 


Region. Studies exploring the effects of career 
academies are limited to the United States. 


8 See Appendix C for a description of the search strategy and search terms. 
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Findings 


Since the mid-1980s, several studies have been 
conducted to evaluate the effectiveness of career 
academies; these have primarily used a matched 
comparison group design. In general, the researchers 
found that career academy students performed 
better than similar (individually matched) students 

in the same high schools, in terms of their academic 
performances and graduation rate (Reller, 1984, 

1985, 1987; Stern, Dayton, Paik, & Weisberg, 1989). 
Then, in 1993, the MDRC (Manpower Demonstration 
Research Corporation) launched a longitudinal 
evaluation study with a random assignment design. 
The MDRC study was conducted in ten different 
career academy sites focusing on preparation for 
various industries, including Business and Finance 
(Pennsylvania and Maryland), Public Service 
(Washington, DC), Aerospace Technology (Florida), 
Travel and Tourism (Florida), Health Professionals 
(Texas), Global Business (Califorina), Video (Califorina), 
and Electronics (Califorina). The initial sample size 
from the 10 sites was 1,952 students, but later the 
sample was reduced to 1,764 students at 9 schools: 
959 students were randomly assigned to the program 
group and 805 students were randomly assigned to 
the control group at the end of 8th or 9th grade. The 
study collected data from school transcript records 
(students’ daily attendance rates, credits earned 
toward graduation, and course-taking patterns); 
student surveys (school experiences, employment and 
work-related experiences, extracurricular activities, 
preparation for college and post-secondary jobs, 

and plans for the future); and standardized math, 
computation, and reading comprehension tests with a 
subsample of 490 students (from both groups). As the 
MDRC longitudinal randomized trial is the most rigorous 
and comprehensive study, its findings have been 
summarized in Table 1. 


Overall findings from all 17 studies (both the earlier 
matching studies and the randomized MDRC studies) 
are discussed below and are grouped by outcome 
measures: academic achievement, post-secondary 
enrollment, and various workplace outcomes. 


Academic performance during high school and 

high school completion. Several studies in the 1990s 
found that academy students performed better than 
similar students in the same high schools; students 
were matched on demographic variables and 9th 
grade records, including grades, high absenteeism, 
and disciplinary problems. Academy students on 
average showed significantly higher outcomes in 
school attendance, credits earned toward high school 
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graduation, grade point average (GPA), and retention 
through high school (Stern et al., 1989). Within the 10 
academies studied by MDRC, however, the impact on 
academic performances varied across sites. Although 
the MDRC sample demonstrated higher academic 
outcomes for career academy students in “completed 
credits to graduate,” they did not show improvement 
on standardized measures of reading and math 
achievements or GPA (Kemple & Snipes, 2000). 
Similarly, academy students at the National Academy 
Foundation (NAF) schools, who were interested in 
finance or the travel industry, did not have higher GPAs 
than their non-academy peers (Orr, Bailey, Hughes, 
Karp, & Keinzl, 2004). 


With respect to the issue of dropping out, the NAF 
report found that 92% of the career academy seniors 
“did not think about dropping out” while only 84% of 
non-academy students “did not think about dropping 
out,” a statistically significant difference. The actual 
dropout rate of the students, however, was not 
reported. Kemple (2008) reported that approximately 
75% of the academy students graduated from their 
high schools on time compared to 65% of similar 
non-academy students from similar school districts. 
Kemple and Snipes (2000) compared the dropout rates 
among academy and non-academy students, and 
there was no significant difference among all sampled 
students (10% for academy students versus 12% for non- 
academy students). 


However, when focusing only on high-risk students 
(based on selected background characteristics 

and prior school experiences), it was found that the 
program might have a larger impact on dropout rates 
(dropout rates were 32% for non-academy students 
versus 21% for academy students). Also, academy 
students showed a significantly higher average rate 
of attendance throughout high school when only the 
high-risk subgroup was considered (attendance rate 
of 76% among non-academy students versus 82% for 
academy students). It seems that participation in an 
academy made a significant impact; however, these 
results should be interpreted with caution, as around 
30% of the students who enrolled in the academy 
programs studied by MDRC left the program before 
graduation, switching to other programs in the same 
high school or moving to different high schools. 
Kemple (2008) suggested, “Academies may be able 
to influence students’ behavior and performance for 
only a year or two before students move on to other 
opportunities” (p. 11). 


In one study, Stern et al. (1989) conducted a cost- 
benefit analysis of the program. To do this, they 
estimated the number of dropouts saved in each 


academy by comparing the number of actual 
dropouts with the number that would have occurred 

if academy students had been included in the 
comparison (non-academy) group. In this sample 
(N=327), it was estimated that 30 dropouts were saved, 
valuated at $41,000 per dropout. With their cost-benefit 
calculation formula, the authors concluded that the 
dropout prevention program for this cohort saved 
between 1 1.3 million dollars.’ 


Enrollment and academic outcomes in post-secondary 
schools. Career academies are expected to produce 
a positive impact on post-secondary school enrollment 
and academic outcomes within schools. Educational 
aspirations for advancement to post-secondary 
education is one of the outcomes investigated in this 
literature (Castellano, Stone, Stringfield, Farley-Ripple, 
& Overman, 2007; Rojewski et al., 2010). Castellano et 
al. (2007) found that educational plans and aspirations 
were higher on average for career academy 
participants. In contrast, Rojewski et al. (2010) did 

not find any long-term differences. The difference 

may be explained by the length of exposure to the 
program or timing of data collection: Castellano et al. 
collected all data while the students were still in school, 
whereas Rojewski collected data both when students 
were in 12th grade and two years after their program 
involvement, suggesting that the effect of career 
academies on future education aspirations might be 
stronger in the early years of high school. A second 
outcome that is frequently examined is the proportion 
of students who enroll in post-secondary education. 
Within the MDRC sample however, no significant 
differences between academy and non-academy 
students were found: the post-secondary enrollment 
rate of academy students was 54.8% versus 54.6% for 
non-academy students (Kemple, 2001, 2008). 


From the late 1990s onwards, Maxwell (2001) and 
Maxwell and Rubin (2002) collected data consisting 

of information on applicants to degree programs at a 
medium-sized state university in California who applied 
from a single high school district. More than 90% of the 
district students were ethnic minorities and nearly 40% 
were from poor families (defined as eligible for free 
lunch). Maxwell (2001) followed the graduates from 
this district to the state university, regardless of whether 
they had attended the academies. They found that 
academy participation in fact negatively influenced 
admittance rates to a 4-year college by 22 percentage 
points. The authors were not able to explain the results, 
but speculated that high school administrators may 
have steered lower-performing students into the 
academies. The authors found some positive impacts: 
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academy students were less likely to need remedial 
course work in English at university and more likely to 
receive a bachelor’s degree by 4.3 percentage points. 


Maxwell and Rubin (2002) extended their analysis to a 
comparison between single district data and national 
data drawn from the National Education Longitudinal 
Study (NELS), to examine if academy participation 
influenced the possibility of attending post-secondary 
schools. The authors found that students in academies 
experienced an 11.6 percentage point increase in 
their probability of attending a 2- or 4-year college 
compared to general track students. Also, participating 
in career academies increased the likelihood that 
students would attend a 4-year college by 

17.9 percentage points. 


Labor market outcomes. A career academy is 
supposed to provide its students with greater career 
awareness through work related courses and work- 
learning internship opportunities. Kemple (2008) found 
that “Academy students were much more likely than 
their non-academy counterparts to be exposed to 
such experiences as job-shadowing, career fairs, 
guest speakers from local businesses, and instruction 

in how to look for and apply for a job, prepare a 
résumé, and interview” (p. 9). One indicator used to 
measure the impact of career academies on labor 
market outcomes is monthly earnings. Kemple (2004, 
2008) surveyed a sample of students approximately 

96 months after their graduation from high school and 
found that on average, career academy graduates 
earned 11% more than non-career academy members 
($2,088 a year, or $16,704 over eight years in total 
earnings). The increase in earnings for the academy 
students was more apparent among young men, while 
the impact was smaller among young women. 


Page (2012) analyzed the MDRC'’s data to examine 
the impact of duration of actual academy enrollment 
on student outcomes, as the actual enrollment length 
varied among program participants. She measured 
the students’ enrollment in the program by the number 
of years and tried to make a causal connection 

to monthly earnings: male students who remained 
enrolled in the academy throughout high school 
subsequently earned approximately $588 more per 
month than non-academy students. More modest 
impacts were found among those who participated 
only partially in the program. 


Castellano, Sundell, Overman, and Aliaga (2011) 
examined student surveys from two districts where 
the students were matched by a propensity score 


9 For detailed calculation, see http://epa.sagepub.com/content/11/4/405.full.odf 
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matching method. The student surveys in one district 
showed that the academy students researched 
potential jobs and careers significantly more than the 
non-academy group (91.8% of academy students 
researched potential jobs versus 86.6% of non- 
academy students); the data was not reported for the 
other district. 


Limitations and Implications 


Based on the findings of the above studies, the career 
academy model has high potential for encouraging 
students to stay in school and preparing them for 
college and the workplace. While a number of studies 
found that career academy students had more 
promising educational outcomes than their peers (in 
the same high schools) in terms of school attendance, 
credits earned toward high school graduation, GPA, 
and retention through high school, a more rigorous 
study (longitudinal RCT) conducted by MDRC found 
that the impact on academic performances varied 
across sites: Although studies in some sites revealed 
that career academy students have higher credit 
completion, students did not show improvement on 
standardized assessments or with GPAs. Additionally, 
though career academy enrollment was associated 
with higher graduation rates, the results on dropout 
rates were mixed. However, when focusing only on 
high-risk students, evidence suggests that the program 
had a positive impact on reducing dropout rates and 
increasing attendance rates. 


Moreover, studies found mixed results of the impact of 
academies on educational aspirations for, admittance 
into, and enrollment in post-secondary education. With 
respect to labor market outcomes, academy students 
benefited from the work-related courses and work- 
earning internship opportunities and earned more 

than non-career academy students, but the increase in 
earnings varied by gender and the years of enrollment 
in a career academy program. However, the schools 
selected in these studies tended to be schools that 
implemented their career academy programs with high 
fidelity, thus these findings might not be representative 
of all career academies. For example, it was found that 
many academies diminished the provision of work- 
related-learning over time, due to the parallel pressures 
to meet the high schools’ academic standards (Visher, 
Altuna, & Stefan, 2013). Thus, it is important to examine 
the context and environment of each individual career 
academy in order to assess its outcomes. 


10 See Appendix C for a description of the search strategy and search terms. 
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through E-Learning 


Katharine Conn 


E-learning programs, modules, and coursework 
offer alternatives to students in technical fields, as 
well as to those in the workforce seeking additional 
qualifications and skills. Such programs provide 
flexibility regarding both the timing and location of 
course content delivery, and can allow students to 
tailor their coursework to their skill set. This section 
explores the efficacy of e-learning programs for 
workforce preparation as well as workforce training. 
As in the previous sections, it includes a description 
of the literature as a whole (program targets, 
publication types, outcomes, methodologies, regions 
covered, etc.), the relevant findings associated with 
this body of work, and any limitations or implications 
of these studies. 


Literature Description 


Twenty efficacy studies in the field of e-learning for 
post-secondary young adults/adults were identified 
for inclusion in this synthesis.'° In addition, two meta- 
analyses and 30 high-quality non-experimental case 
studies, journal articles, and policy reports were 
included. The following section provides an overview 
of the characteristics of the studies found through the 
systematic review process. 


Study Population. Of the 20 e-learning efficacy 

studies identified for this synthesis, 8 were targeted 

at medical students and health professionals, 3 were 
targeted at university science/ psychology students, 

2 sampled graduate students in education or pre- 
service teachers, 5 targeted vocational education 
students and industry technicians, 1 sampled 

business school students, and 1 employed a sample 

of community college students. While this sample of 
rigorous studies may seem unduly weighted to students 
in the medical field, this is partially reflective of the 
number of e-learning programs available in the field of 
medicine currently, as well as a tendency for the use 
of randomized trials for research within the medical 
sciences. 


Publication Type. Of the 20 efficacy trials identified 
for this synthesis, 18 are journal articles, 1 is a working 
paper, and 1 is a report. 


Outcomes. The main outcome examined by this 
group of efficacy studies is student achievement 

or performance. The majority of studies provided 

an estimate of the average treatment effect, while 

a limited number assessed the impact on sample 
subgroups (low-income or low-performing students). 
Secondary outcomes include student satisfaction and 
attitude. 


Methodology. Of the 20 e-learning-related efficacy 
studies, 18 employed the use of randomized controlled 
trials, 1 employed a matching strategy, and 

1 employed an instrumental variables strategy. 


Region. The majority of efficacy literature on e-learning 
is from the United States. Out of the 20 efficacy studies, 
14 were conducted in the United States, 2 in Asia, 

2 in Europe, one in the Middle East, and one in Latin 
America. 


Findings 


Based on the efficacy literature above, this section 
synthesizes the findings of these studies into two major 
themes: the lack of significant achievement differences 
between e-learning and traditional classroom teaching 
and the heterogeneity within these findings. 


Lack of significant achievement differences. Of the 

20 experimental and quasi-experimental efficacy 
studies included in this synthesis of e-learning in post- 
secondary education or the workforce, the majority 
(15) found that there was no statistically significant 
difference between in-class, traditional learning and 
online or computer-based learning in terms of student 
performance; in each case, the authors could not 
reject the null hypothesis that the means of each 
group (treatment and control), were the same. In 
addition, 4 studies found slightly positive and significant 
effects of online learning on achievement (versus a 
control group) (Schilling, Wiecha, Polineni, & Khalil, 
2006; Schoenfeld-Tacher, McConnell, & Graham, 
2001; El-Deghaidy & Nouby, 2008), while one study 
found a negative and statistically significant impact 
of e-learning on achievement in a community college 
context (Jaggars & Bailey, 2010). 


The absence of a measurable difference in 
achievement between the two groups in the majority 
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of studies (or even a slight advantage to the online 
group) held among a variety of young adult and adult 
learners: medical students and health professionals 
(Bello et al., 2005; Beyea, Wong, Bromwich, Weston, 

& Fung, 2008; Benjamin et al., 2008; Harris et al., 

2007; Hugenholtz, de Croon, Smits, van Dijk, & 
Nieuwenhuijsen, 2008; Padalino & Peres, 2007; Schilling 
et al., 2006; Schoenfeld-Tacher et al., 2001); business 
school students (Ocker & Yaverbaum, 1999); university 
science and psychology students (Frederickson, Reed, 
& Clifford, 2005; Sexton, Raven, & Newman, 2002; 
Woottipong, 2013); graduate students in education 
and pre-service teachers (El-Deghaidy & Nouby, 

2008; Mentzer, Cryan, & Teclehaimanot, 2007); and 
vocational education students and industry technicians 
(Benson et al., 2004; Day, Raven, & Newman, 1998; 
Gilliver, Randall, & Pok, 1998; Hairston & Nafukho, 2011; 
Inayat, ul Amin, Inayat, & Salim, 2013 ).'' Further, this 
overall “no significant difference” trend seemed to 
hold, regardless of the length of the training sessions (as 
little as four weeks up to a full calendar year). 


For example, Benson et al. (2004) used a quasi- 
experimental matching methodology (and sample 
population of 193 career and technical education 
students in three community colleges across three 
states in the United States) and found no significant 
difference in student achievement between online 
and campus-based delivery models. However, the 
authors provided limited evidence of balance among 
observable individual or course characteristics post- 
matching, casting doubt on the equality of these 
groups. Day et al. (1998) randomized 58 students in 

a technical writing course (in agri-communication) 
between three traditional lectures per week versus a 
blended learning course with two media-enhanced 
lectures plus a computer laboratory session. They found 
no significant differences between the achievement or 
attitudes of students in the web-dependent versus the 
classroom model. Inayat et al. (2013) used a pre-post 
achievement gains methodology in an evaluation of 
online and in-class programs in graphic/web-design at 
a vocational training institute in Pakistan; they found 
that for two courses, there was a statistically significant 
increase in performance but that in two other courses 
there was no such pre-post difference. One study 
evaluated the use of e-learning in the workplace 

itself; Hairston and Nafukho (2011) conducted a 
randomized controlled trail of 262 participants across 
six industries (retail, manufacturing, civil service, 
architectural, education, and trucking) and found that 
while the e-learning groups make larger performance 
gains, the difference was not statistically significant. 
Furthermore, Gilliver et al. (1998) evaluated an 


11 This study employed a pre-post methodology, not an experimental or quasi-experimental methodology. 
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e-learning course in financial accounting among 

151 students at Singapore’s Ngee Ann Polytechnic 
through a randomized-controlled trial and found 

that on average, there was no advantage to online 
learning; however, for students at either end of the 
ability distribution (low and high performers), there was 
a positive impact on learning, presumably due to the 
fact that the web-based learning platform allowed 
students to personalize their instruction by learning 
level and alter the pace of the work. 


However, Xu and Jaggars (2013), in a quasi- 
experimental evaluation of online (versus face-to-face) 
programs across 34 community colleges nationwide, 
found robust negative estimates for online learning in 
terms of both course persistence and course grade. 
The authors’ analyses further suggested that previous 
descriptive estimates of the impact of these e-learning 
programs may in fact underestimate the gap in 
performance of traditional versus online students, 

as online students in this study were arguably better 
prepared and more motivated than those that took 
traditional courses; thus the direction of the effect size 
bias may be in the opposite direction of what was 
previously thought. 


Overall findings may mask heterogeneity. This 
“absence of a statistically significant difference” 
between online and face-to-face learning (in regards 
to achievement) was a conclusion also reached by a 
recent meta-analysis of e-learning studies with sample 
populations from kindergarten up to workforce training 
(United States Department of Education, Office of 
Planning, Evaluation, and Policy Development, 2010). 
However, Jaggars and Bailey (2010), in a response 

to the Department of Education, re-examined these 
studies and found that the primary driver of these 
results are studies that employ blended learning 
programs (and not fully online courses alone). Further, 
they found that among students who completed their 
online courses, achievement levels were comparable 
to students in face-to-face classrooms; however, 
online students were more likely to drop out of their 
courses. Further, Sitzmann, Kraiger, Stewart, and Wisher 
(2006), also used meta-analytic techniques to show 
that web-based training was more effective than 
classroom instruction only in web-based training where 
the trainees had control of the pacing, were able to 
practice, received feedback during training, and were 
in longer courses. Finally, the United States Department 
of Education (2010) found that online instruction 

that was collaborative or teacher-guided was more 
effective than independent online instruction; of these, 
they found that teacher-guided instruction had the 
highest pooled effect size (0.039 standard deviations). 
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While the efficacy evidence of e-learning programs 
focuses largely on post-secondary populations in 
school or the workforce (and not unemployed workers), 
the evidence available suggests that on average, 
e-learning programs can be just as effective as face- 
to-face programs. However, the overall finding of “no 
significant differences between groups” may mask 
heterogeneous effects: there are poorer results and 
higher dropout rates for lower income online students 
(Jaggars & Bailey, 2010) and potentially high impacts 
for students on either end of the ability distribution due 
to highly-personalized e-learning software (Gilliver et 
al., 1998). Further, meta-analytic evidence suggests 
that blended learning programs (and not fully online 
programs), as well as e-learning programs that involve 
extensive practice, feedback, and time commitments 
are key to any online learning advantage (Jaggars & 
Bailey, 2010; Sitzmann et al., 2006); otherwise classroom 
instruction still prevails. Further, for courses that train 
students to interact with others (through teaching or 
business negotiations), student satisfaction was higher 
in face-to-face courses (Mentzer et al., 2007; Ocker 

& Yaverbaum, 1999). Finally, online instruction that 
was collaborative or instructor-led was overall more 
effective than individualized learning (United States 
Department of Education, 2010). 


These preliminary findings imply: 
e = Any e-learning training should pay careful attention 
not only to the average treatment effects but 
also to the progress made by multiple subgroups 
(by income-level, ability-level, etc.). In addition to 
learning, outcomes such as sample attrition in both 
groups should also be carefully monitored. 


e E-learning programs should incorporate elements 
of instructor-led/blended learning (as opposed to 
fully online courses). Programs should also include a 
collaborative element. 


Policy-relevant considerations. In addition to the 
efficacy literature (above), this review also includes a 
series of policy-relevant considerations for e-learning 
in Technical and Vocational Education and Training 
(TVET), based on academic descriptive or case study 
research and a series of UNESCO, the World Bank, and 
OECD reports. This set of considerations is not meant 
to be comprehensive, though they do encompass 

the major themes of recent policy-relevant research. 
The following paragraphs explore the need for ICT skill 
development, strategies for e-learning take-up, the 
quality of the pedagogical content, and the costs 
associated with implementing these e-learning training 
programs. 


Need for Information and Communication Technology 
(ICT) skills development. Multiple studies raised 

the issue of technical skills within both the student 
population and the teaching population that 

may need to be addressed before instituting any 
Information and Communication Technology (ICT) 
programs within vocational or workforce training. For 
example, Jinnah, Al-Mamum, Khan, and Hasan (2011) 
conducted a survey regarding the overall status of ICT 
in the Polytechnic Institutes of Bangladesh and found 
that both teachers and students lacked skills that would 
enable them to use ICT for their learning purposes 
due both to the poor quality of ICT infrastructure and 
to the poor ICT preparation of teachers. In addition, 
UNESCO (2013) found that a main concern in the 
implementation of ICT in TVET was that teachers 

were often not prepared to use ICTs in teaching and 
require additional training and targeted professional 
development in how to incorporate technology for 
improved educational outcomes. Further, Goertz and 
Johanning (2004), Bedwell and Salas (2010), and Dunn 
(2003) all emphasized the need for an initial needs- 
assessment (for students, teachers, or employees) to 
better understand the individual computing levels, 
knowledge, and skills of all stakeholders before the 
implementation of any IT-based trainings. 


Strategies for e-learning take-up. Various authors have 
offered suggestions regarding strategies to encourage 
the take-up of e-learning programs in the workforce. 
Among the challenges to the integration of IT into 
technical training in the South African manufacturing 
workforce was the need for employers to set aside 
adequate study hours for e-learning (Macdonald, 
Bullen, & Kozak, 2010). This finding is echoed by Goertz 
and Johanning (2004) who also suggested that some 
degree of time-sharing (employers allot one hour per 
week of work time, while employees allot one hour per 
week of their leisure time) may be optimal. Goertz and 
Johanning (2004) also suggested that take-up could 
be increased though offering employees incentives, 
ensuring that the technical system is stable (to avoid 
frustration and drop-out), and through employing tutors 
and learning groups. Finally, Karaali, Gumussoy, and 
Calisir (2011) used structural equation modeling to 
determine factors that predict take-up of e-learning 

in the workplace, and they found that above all other 
variables, social influence has the highest predictive 
power, highlighting the importance of creating a 
workplace culture that is supportive of continuous 
professional development through web-based services. 
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Quality of pedagogical content. In the same way 
that an individual teacher can drastically influence 
the learning levels of students (in either direction), 
individual e-learning programs can also have such 
impacts. That is, e-learning programs are not effective 
in and of themselves, nor are they “ready-made,” but 
instead they rely on specific pedagogical qualities 
and must be adapted to each context in order to be 
successful (Clark & Mayer, 2013; Bedwell & Salas, 2010; 
UNESCO, 2013). Curtain (2002), Dunn (2003), Bedwell 
and Salas (2010), and Clark and Mayer (2013) stressed 
the “importance of interactivity” for online learning 
and blended learning, in terms of both interactive and 
adaptive e-lessons as well as the interactivity of the 
students with other students and instructors. Mupinga 
and Busby (2011) highlighted a series of web-based, 
interactive simulation trainings used to improve 
students’ technical skills.'2, Regarding teacher-student 
interactivity, Clark and Mayer (2013) highlighted 
evidence that points to the importance of “guided 
discovery” as opposed to fully unguided discovery 
learning online. And UNESCO (2013) cited growing 
agreement among all stakeholders (vocational 
schools, government training programs, and industries) 
that “ICTs could and should be incorporated into 
TVET, although they are unlikely to replace in-class 
instruction” (p. 14). There is also a growing debate 
regarding the “degree of ‘learner control’” (pace 
control, sequence control, content control, and 
advisory control) provided to the students, and may 
depend on the nature of the course; Granger and 
Levine (2010) found that increased “learner control” 
can be advantageous, but only within a certain realm 
of course complexity. 


In addition, Bedwell and Salas (2010) argued that there 
are a series of “learning principles” that are critical 

for web-based training success (as well as face-to- 
face learning). Among these are the importance of 
gaining the attention of the e-learning participant 

with an engaging introduction, delineating lesson 
objectives and activating motivation, initiating recall 
of prior learning, presenting stimulus material, providing 
learning guidance, eliciting performance, providing 
feedback, assessing performance, and enhancing 
retention and learning transfer. Clark and Mayer (2013) 
further argued that there needs to be a certain degree 
of learner scaffolding throughout any web-based 
lesson. 


12 Examples of web-based simulation games aimed to teach technical skills include: Building Homes of Our Own; DimensionM Game Maker 8; Klik & 


Play; MS Flight; Simulator; Ruler game; West Point Bridge Design. 
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Costs associated with instituting ICT in TVET. Numerous 3. “There is great potential for job creation and 
authors cite the need for countries, schools, and economic growth to emerge from improved TVET 
industries to accurately assess the costs associated and a knowledge-based economy if the 

with any e-learning training platform. While there are benefits can be extended to wider parts of the 


certain economies of scale associated with large population” (p. 15). 


e-learning initiatives, the upfront costs associated 
with the creation of the programs can prove to be 
extremely high; costs to be considered include the 
software platform, up-front licensing fees, IT system 
administration and learning management system, 
pedagogical content (and support materials), 
information communications infrastructure, the initial 
training of teachers and students, and a supervisory 
organization and credentialing network (Paulsen, 
2009; Bedwell & Salas, 2010; Moran & Rumble, 2004; vecational or community college degrees: but would 
instead begin to prioritize “competency-based” e-learn 
modules (associated with particular technical skills). 
The authors argued that this type of “competency- 
based education” would allow students or employees 
to complete their training over their lifetime, continually 
forming a “stack of credentials,” that can make them 
more marketable in the labor force. 


McKay and Vilela (2011) stressed the importance of 
continued evaluations of any e-learning program 
(they examined several training effectiveness models) 
in order to improve upon the content as well as the 
implementation. Finally, Weise and Christensen (2014), 
predicted that the future of higher education in the 
United States would be less focused on traditional 
academic routes, or even on the one- to two-year 


Dunn, 2003). Large-scale ventures in ICT for TVET in 

the UK (Open Tech) and across Africa (African Virtual 
University) have proved difficult to sustain and grow 
due largely to these cost factors (Moran & Rumble, 
2004). This is particularly true for small and medium- 
sized enterprises (Paulsen, 2009). Further, Paulsen 
(2009), whose recommendations were derived from 
case studies of eighteen enterprises in eight European 
countries, suggested that small-sized enterprises with 
few employees may prefer to enroll in more generic 
e-learning courses, while medium-sized enterprises 
could benefit from more specialized courses (offered 
through branch associates or in partnership with 
suppliers or chains), whereas as large enterprises could 
afford to develop their own e-learning courses, tailored 
for internal use. 


Benefits to ICT in TVET. While there are a number of 
important considerations which could affect the 
timing, scope, and success of an e-learning training, 
particularly in the developing world, the research and 
policy literature is still cautiously optimistic about the 
potential for ICT to transform vocational and technical 
training in the workforce. In particular, UNESCO (2013), 
found that: 


1. “Good knowledge of ICTs combined with the 
resulting development of soft and transferable 
skills, in addition to the technical training provided, 
stands to benefit all students and teachers in the 
constantly changing jobs marketplace of today, 
forming a toolkit for long-term employability and 
success in the workforce” (pp. 14-15). 


2. “The use of ICTs could improve both the quality 
and the perception and status of TVET, which is not 
currently held in high regard in many countries” 

(p. 15). 
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A second solution domain offering strategies to 
strengthen the technical workforce is composed of 

two main strategies that seek to improve access to 
career and technical education: the provision of better 
information to students, their parents, and prospective 
employees; improvements in career counseling and 
employment services; and the provision of incentives 

to students in order to encourage participation or 
enrollment in a particular field. 


This chapter first explores studies that examine 

the impact of providing information to students or 
parents regarding the returns to certain types of 
education, their eligibility for financial aid, or about 
the performance of schools in their choice set. In a 
knowledge-based economy, such information has 
the potential to greatly impact enrollment outcomes. 
Secondly, this chapter details the impact that 
information provision through career counseling and 
employment services can have on an individual’s 
occupational choices; these studies examine the 
impact of career counseling and services on both 
subjective measures (career initiative, self-efficacy, 
and satisfaction) and objective measures (salary, 
career advancement/promotion, and employment). 
Lastly, this chapter examines the ways in which 
monetary incentive structures directed at students 
(scholarships, vouchers, etc.) can stimulate 
participation in education or skills training; the studies 
reviewed here explore the overall impact of monetary 
incentives on student outcomes, the impact when 
performance is incentivized, and the impact on 
specific groups including underprivileged subgroups 
and females. 
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Information Interventions 
Katharine Conn 


While information interventions in this area are similar 
in that they focus on drivers of educational choice, 
the type of information provided in each case 

can be quite different. In this review, information 
provided to students and households includes: the 
rate of return to a particular educational investment, 
information on aid-eligibility, and information on 
relative school performance. Note that these 
information interventions were temporary in nature 
and not permanent features of an education 
system. This section explores the efficacy of these 
information interventions. It includes a description 

of the literature as a whole (study population, 
publication types, outcomes, methodologies, regions 
covered, etc.), the relevant findings associated with 
this body of work, and any limitations or implications 
of these studies. 


Literature Description 


Nine impact evaluations dealing with the effects of 
information provision on educational choices are 
synthesized here. The following section provides an 
overview of the characteristics of the studies found 
through the systematic review process and which are 
included in this synthesis. 


Study Population. Of the nine studies that examined 
the impact of information on school choice, eight 
were targeted at students currently attending either 
primary or secondary school, and one sampled out-of- 
school youth. 

Publication Type. Of these nine studies identified for this 
synthesis, three are working papers and six are journal 
articles. 


Outcomes. Studies that examined the impact of 
information provision on school choice examined 
outcomes such as school completion, dropout rates, 
attendance, academic performance, hours worked, 
submission of admissions and aid applications, and 
enrollment/attainment. 


Methodology. Of these nine information studies, eight 
employed randomized controlled trials, and one 
employed a descriptive survey-based methodology. 


Region. These information interventions took place 
in China (two), the Dominican Republic, the United 
States (three), Madagascar, India, and Canada. 
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Findings 


Based on the studies above, this section synthesizes the 
results into two major themes: 1.) the impact of rate 

of return information on educational choice and 2.) 
the impact of aid eligibility and school performance 
information on educational choice. 


Impact of rate of return information. Seven studies 
examined the following question(s): What happens 
when individuals are informed about the financial 
returns or the benefits of a certain type of education? 
Does this affect their school choice? Jensen (2010) 
conducted a randomized trial in which students (8th 
grade males) at randomly selected schools in the 
Dominican Republic were informed (by the survey 
enumerator) of the differential rates of return to 
primary, secondary, and tertiary education. He found 
that this information caused students in the treatment 
groups to complete on average 0.20-0.35 more years 
of school over the next four years than those who did 
not receive this information. Similarly, Nguyen (2008) in 
Madagascar, conducted a randomized trial in which 
teachers relay information to parents and 3rd/4th 
grade students about the true returns to schooling. 
Nguyen found that test scores improved by 

0.37 standard deviations for those whose initial 
perceived returns were under the rate of return 
statistics, while student attendance in intervention 
schools also increased by 3.5 percentage points. 
Nguyen also experimented with the use of “role 
models” and sent a role model to share with students 
and families his/ her family background, educational 
experience, and current achievements. He found that 
this intervention was particularly impactful for poor 
students who were sent role models of initially poor 
backgrounds. 


Adding more evidence to the theory that correcting 
an individual's perception of the returns to education 
can influence actions or beliefs, Oreopoulos and 
Dunn (2013) conducted a randomized experiment in 
Canada in which the authors presented students with 
an online survey and randomly sampled half of the 
students to receive a three-minute video within the 
survey about the benefits of post-secondary education 
(these students were also invited to try out a financial- 
aid calculator). Students who were selected for the 
video reported higher expected economic returns, 
lower concerns about costs, and expressed a greater 
likelinood of continuing their education past the 
secondary level. 


Finally, Mensch, Grant, Sebastian, Hewett, and 
Huntington (2004) examined the impact of an 


intervention in which reproductive health information, 
vocational counseling and training, and assistance 
with opening savings accounts were provided to 
adolescent girls in slum areas of Allahabad in Uttar 
Pradesh, India. The vocational counseling portion of 
this intervention put an emphasis on the importance 
of remuneration and savings, and consisted of the 
use of flip books that contained vocational flash 
cards with a description of courses available. The 
counselors (peer educators) also provided detailed 
information about short-term, non-formal training 
courses available in the vicinity, including Courses 
offered by government institutes and NGOs, as well 
as those organized specifically by the project. While 
take-up of these vocational programs was very high, 
the authors reported that there were no statistically 
significant effects of the program on the number of 
hours women spent in the labor market or at training 
sessions. This lack of clear findings is likely due to the 
fact that the authors were only able to include a small 
subset of the treated participants in the analysis, as 
they were unable to clearly identify (and follow-up 
with) individuals that took part in the training. 


Thus, while there is suggestive evidence that sharing 
information about the returns to a certain level of 
education may affect schooling decisions or beliefs, 
this is yet to be rigorously tested in the case of 
vocational education. A new study (Hicks et al., 2011) 
is examining this particular issue, but only preliminary 
results are available thus far. In this study, the authors 
paired a randomized evaluation of a vocational 
education voucher program with an information 
intervention in Kenya, in order to test whether the 
provision of information on the returns to vocational 
education (estimated in a Mincerian regression), would 
have an effect on voucher take-up (note information 
was presented verbally as well as through handouts 
and posters). At the outset, the authors found that the 
perceived rate of return to vocational education was 
quite high (61%, compared to the actual value of 37%). 
The authors found that this information intervention did 
not affect individual decisions to apply for a voucher 
or voucher take-up, though it did have an impact of 
the preference of females for more male-dominated 
courses (for more details on this gender component, 
see Chapter | of this paper, Vocational Education, 
Section B, “Differential Impact by Gender.”). 


Finally, in rural China, a randomized information 
intervention at the junior high school level was 
conducted in which students were provided with 
information regarding the income returns to high 
school education as well as counseling regarding the 
requirements for entering high school (Loyalka et al., 
2013a). The authors found that information on income 
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returns had no significant impact on student choice 
(this may be due to the fact that the rate of return to 
secondary education was no different from the rate 
already perceived by the students; the authors do not 
test this in their baseline survey). They did find, however, 
that counseling had a negative impact on student 
achievement and transition into high school: the 
authors surmised high school, in combination with the 
relatively high wages available in the unskilled sector. 


Impact of aid eligibility and school performance 
information. Three additional randomized trials 
examined the impact of providing information 

on post-secondary aid eligibility and on school 
performance in order to influence student educational 
choices. Hastings and Weinstein (2008) conducted 

a randomized experiment (and also used a natural 
experiment) that provided direct information about 
school performance (school test scores) to low-income 
parents under a public choice plan in the United 
States. They found that receiving such information did 
increase the probability that parents would choose a 
higher-performing school for their children, and that 
their children in turn would improve their academic 
achievement, which supports the hypothesis that the 
provision of performance information (which could 

be considered a form of “return on investment”) can 
indeed have an impact on school choice. 


Secondly, in a randomized experiment in which H&R 
Block tax professionals provide student aid eligibility 
information as well as assistance to low- to moderate- 
income families (in the United States) in the completion 
of the FAFSA, Bettinger et al. (2012) found that students 
who received this information and assistance were 
substantially more likely to submit the aid application, 
enroll in college the following fall, and receive more 
financial aid. Interestingly, these results did not hold 
for a secondary treatment group who received only 
aid eligibility information alone, highlighting the 
importance of in-person assistance. In addition, Kane 
and Avery (2004) gathered survey data on student 
perceptions of college opportunities, particularly 
among low-income students and found that the 
aspirations and expectations of these students are 

not overly pessimistic, thus interventions that don’t go 
beyond providing information regarding aid eligibility 
(as well as information on other returns) may not prove 
to be sufficient to change the application behavior 

of low-income youth in the United States (study 
conducted in Boston public schools). These results 
support the findings of Bettinger et al. (2012) in that 
interventions that go beyond the simple provision of 
information may be the most compelling to youth, at 
least within the United States context. 
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Finally, Loyalka, Song, Wie, Zhong, and Rozelle (2013b) 
conducted a randomized trial in rural China in which 
they provided high school students with information 
about college costs and financial aid opportunities. 
They found that this intervention raised the probability 
of students receiving some type of financial aid, 
particularly among females and low-income students. 
The intervention also increased the likelinood of college 
attendance, especially among females, raising the 
probability of female attendance by 10.7 percentage 
points, but with no significant effect on male students. 
However, the intervention did not seem to impact the 
type of college to which students applied (tier 1 and 2 
versus others). 


Online provision of occupational information. While 
there is no efficacy literature evaluating the impact 
of online employment websites or networks rigorously, 
there are organizational websites (publicly as well as 
privately run), which provide information across all 
categories of employment, including certain websites 
that specialize in employment positions with STEM 
fields, for example, the International Association for 
Educational and Vocational Guidance, Australian 
Apprenticeships, and Asia Pacific Career Development 
Association, among others. (see Appendix B for a 
complete list and accompanying websites). 


Limitations and Implications 


These recommendations are drawn only from the 
small sample of impact evaluations discussed 

above, but from this limited evidence across multiple 
countries, it seems that informing students of the 
returns associated with various levels of education 
can be effective in influencing school choice (Jensen, 
2010; Oreopoulos; Nguyen, 2008; Mensch et al., 2004; 
Hastings & Weinstein, 2008; Loyalka et al., 2013a; 
Loyalka et al., 2013b). Clearly, the nature of the 
information intervention may depend greatly on the 
country context and student population itself. Further, 
the extent to which these interventions may actually 
influence change partially depends on the degree of 
misperception in students’ appraisal of income returns 
associated with a certain level of education. In Kenya, 
where individuals had overly optimistic perceptions of 
the rate of return to vocational education, it was not 
surprising that the provision of true returns information 
had no statistically significant impact on student 
demand for vocational education (Hicks ef al., 2011). 
This is in contrast to the Dominican Republic where the 
rate of return to education was perceived as lower 
than its true value by the students in the intervention 


13See Appendix C for a description of the search strategy and search terms. 


Consortium for Policy Research in Education | cpre.org 


sample, and the impact of an informational 
intervention had a positive and statistically significant 
impact on schooling attainment (Jensen, 2010). In 
addition, the success of information interventions may 
also depend on the level of information technology 
and thus information dissemination mechanisms 
available locally. Finally, counseling and guidance 
regarding the options open to adults, the returns 

on those options, and assistance with filling out 
complicated forms, may generate an even larger 
impact on individual choices (Bettinger et al., 2012; 
Kane & Avery, 2004). The next section provides a 
more in-depth review of such career counseling 
interventions. 


Career Counseling and 
Employment Services 


Elizabeth Park 


The provision of information alone may not 

be sufficient to affect career choice or labor 
market outcomes such as salary, promotions, and 
employment. Career counseling, mentoring or 
formal employment services for both students and 
individuals already in the workforce can help to 
bridge this gap. This section explores the efficacy 
of career counseling and employment services. It 
includes a description of the literature as a whole 
(program targets, publication types, outcomes, 
methodologies, regions covered, etc.), the relevant 
findings associated with this body of work, and any 
limitations or implications of these studies. 


Literature Description 


Eighteen efficacy studies regarding the impact of 
career counseling on youth and current employees 
and professionals were identified for inclusion in this 
synthesis.'? The following section provides an overview 
of the characteristics of the studies found through the 
systematic review process and which are included in 
this synthesis. 


Study Population. Of the 18 high-quality studies of 
career guidance identified for this synthesis, only 4 of 
them contained an intervention that was targeted 
at students currently in secondary school or in higher 
education: Fouad (1995) and Kerr and Robinson 


Kurpius (2004) focused on American secondary school 
students; Stoeger, Duan, Schirner, Greindl, and Ziegler 
(2013) focused on female German secondary school 
students; and Wu, Tsai, and Chen (2014) focused 

on students enrolled in 12 universities across Taiwan. 
The majority of the remaining interventions targeted 
currently employed and/or unemployed adults. 


Publication Type. Of the 18 studies identified for this 
synthesis, 13 are journal articles, 3 are reports from 
international organizations, 1 is chapter from an edited 
book, and 1 study is a doctoral dissertation. 


Outcomes. Many of the outcomes examined in the 
career counseling literature focused on subjective 
returns to counseling and mentoring (¢€.g., career 
initiative, self-efficacy, commitment, satisfaction). 
However, several studies addressed the impact of 
career counseling on objective success (e.g., salary, 
promotions, employment). 


Methodology. Of the career guidance-related studies, 
2 employed randomized controlled trials, 2 employed 
propensity score matching, 3 employed comparisons 
across nonequivalent groups, 9 employed correlations, 
and two synthesized findings from other evaluations. 


Region. Of these 18 high-quality mentoring studies, 

2 synthesized evaluations took place across the globe, 
4 took place in Asia (in China and Taiwan), while the 
other 12 took place in America and Europe (including 
France, Germany, Romania, Switzerland, the United 
Kingdom, and the United States). 


Findings 


The career counseling and employee services-related 
studies identified for this synthesis generally reported 
that mentoring has a positive if not null impact on 
career initiative, self-efficacy, and satisfaction. 

With regard to the objective measures, most studies 
reported that mentoring increases earnings and 
promotes career advancement, but the results were 
mixed for employment. 


Subjective Measures. A number of studies, explored 
in more detail below, examined the impact of career 
counseling and services on a series of subjective 


measures, which include: career initiative, self-efficacy, 


and career satisfaction. 


Career Initiative: The career counseling-related 

studies identified for this synthesis tend to report that 
counseling has a positive if not null impact on career 
initiative. Stoeger et al. (2013) evaluated a one-year 
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e-mentoring program focused on promoting girls’ 
development in STEM and its effects on a number of 
outcomes relevant to potential future STEM careers: 
STEM activities, knowledge about university studies 
and jobs in STEM, and academic elective intentions. 
Although the results suggested that there was a 
positive relationship between career guidance and the 
outcomes described above, the short-term (6 months) 
initial increase in STEM activities was not maintained to 
the end of the program. One possible reason Stoeger 
et al. offered is that there is a typical decline in user 
activity after the initial phase of participation in online 
programs. During an initial phase in which subjects are 
getting to know their online community, participation 
rates are typically quite high. However, the rates 
normally fall precipitously thereafter. 


Similarly, in a study on the self-efficacy of secondary 
school-aged students, Kerr and Robinson Kurpius (2004) 
conducted an analysis of an intervention called the 
Talented At-Risk Girls: Encouragement and Training for 
Sophomores, or TARGETS, project, which was created 
to enhance the career development of minority and 
low-income girls who had strong academic potential 
in math and science but who were at risk'* of not 
achieving their career goals. The results suggested that 
counseling interventions positively impacted career 
aspirations and increased career search behaviors 
that were likely to increase students’ knowledge about 
careers in mathematics and science. 


An earlier study by Fouad (1995) explored a one-year 
intervention that was designed to improve 8th grade 
minority students’ awareness of and preparation 

for math and science careers. They found that the 
program was successful in increasing students’ 
occupational knowledge and providing students with 
the knowledge to make deliberate decisions about 
high school. Although across all students there was a 
downward trend in academic achievement over the 
course of the year, in comparison to the control group, 
students who received counseling performed better in 
math and science courses than their counterparts. 


Self-efficacy: While these studies offered very mixed 
definitions of self-efficacy and self-esteem, in general 
most studies reported that career guidance increased 
self-reported measures of ability and value. According 
to Fouad (1995), 8th graders who received counseling 
that infused math and science career awareness 
demonstrated greater self-efficacy after completing 
the treatment. Another study similarly found that for 
secondary school-aged girls, mentoring improved 
self-efficacy; the girls demonstrated higher scores on 


14Risk factors might include low self-esteem and self- efficacy, poverty, unsafe behaviors, or poor family support for goals. 
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self-efficacy measures between the pre- and post- 
tests (Kerr & Robinson Kurpius, 2004). Similarly, Wu, Tsai, 
and Chen (2013) also found that career guidance 

had a positive impact on science education majors 

in Taiwan. The results suggested that the appropriate 
career guidance had a strong positive impact on the 
vocational self-concept (b = 0.58, p < .001) and self- 
perceived employability (b = 0.31, p < .001) among the 
science education undergraduates. 


Although analyzing a different sample of professionals, 
Brennan (2009) also found that active-duty Coast 
Guards who received counseling have significantly 
higher levels of career decision-making self-efficacy— 
including one’s level of confidence of successfully 
engaging in tasks associated with making a career 
choice—as compared to the control group. In 
addition, Chang and Feng (2014) reported that career 
counseling strengthened the positive relationship 
between career competencies and subjective career 
success (defined as career growth and marketability) 
for employees of manufacturing companies in Taiwan. 
Similarly, Bernaud, Gaudron, and Lemoine’s (2006) 
study of adults in the technical training, administration, 
trade, and communication industries suggested that 
the effects of career counseling appeared positive 

for self-efficacy. However, the correlation between 
counseling treatment intensity and the progress made 
in enriching self-concept by the participants were very 
slight and not significant. In contrast, Siebert (1999) 
found no association between mentorship and self- 
esteem among newly hired engineers overall. However, 
he did find that among individuals participating in the 
mentor program the intensity of the mentorship was 
positively related to self-esteem (b = 0.46; p < 0.05). 


Satisfaction: Overall, studies found a positive 
association between career counseling and career 
satisfaction. Despite variations in geographic location 
and industry, the studies included in this synthesis 

all reported that protégé and mentor experiences 
positively influence job satisfaction (Bernaud et al., 
2006; Collins, 1994'S; Seibert, 1999; Pan, Sun, & Chow, 
2011; Gong, Chen, & Yang, 2014). For example, 
based on the written responses from individuals in 
the technical training, administration, trade, and 
communication industries in France, Bernaud et al. 
(2006) found that for individuals who received career 
guidance, fulfillment at work seemed to be the most 


positive gain: 73% of the participants reported greater 
satisfaction at work since completing the career 
guidance process. Furthermore, according to Seibert’s 
(1999) study of newly hired mechanical and electrical 
engineers at a Fortune 100 company, the intensity of 
the mentorship was associated with career satisfaction: 
higher levels of mentoring received by the protégé was 
associated with higher levels of job satisfaction 

(b = 0.34, p < 0.05). 


Addressing a different research question, Ragins, 
Cotton, and Miller (2000) explored whether satisfaction 
with mentoring is related to career attitudes. They were 
able to obtain a nationally representative random 
sample including social workers, engineers, and 
journalists. Among their findings were that protégés 

in highly satisfying mentoring relationships reported 
significantly greater job satisfaction than non-mentored 
individuals. 


Objective Measures. The following section explores 
the impact of mentoring and career services on 
the following objective measures: salary, career 
advancement or promotion, and employment. 


Salary: Although the majority of the studies included 

in this synthesis focused on subjective measures 

of career success, a few explored the relationship 
between mentorship and objective measures such 

as employment, career advancement, and salary 
increases. Although as reported earlier Chang and 
Feng (2014) found that career counseling strengthened 
the positive relationship between career competencies 
and subjective career success, counseling did not 
have a significant positive effect on the relationship 
between career competencies and objective career 
success, measured by salary in this study. Likewise, 
Collins (1994) found that involvement in mentor- 
mentee relationships did not have a significant impact 
on the income level of those who had been a protégé; 
however, it did in fact have a significant impact on the 
income level of mentors.'® '7 


Promotion: Career advancement is another objective 
measure of the effectiveness of career guidance 
programs. For instance, in their evaluation of 
marketing, administration, and finance employees of 
Taiwanese companies, Gong et al. (2014) found that 


15 Professionals who had been both a protégé and a mentor had the highest level of career satisfaction, followed by those who had been a protégé 
but not a mentor, those who had not been a protégé but had been a mentor, and those who had been neither a protégé nor a mentor. 

16 Professionals who had been both a mentor and protégé earned the highest incomes, followed by those who had been a mentor but not a protégé, 

those who had been neither a protégé nor a mentor, and those who had not been a mentor but had been a protégé. 

17 Other studies that have explored the effect of being a mentor on career success found that becoming a mentor benefits one’s own career 

from an objective perspective (e.g., hierarchical advancement and monetary rewards) (Collins, 1994; Liu, Liu, Kwan & Mao, 2009; Bozionelos 


Bozionelos, Kostopoulos & Polychroniou, 2011). 
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guidance benefited protégés because it increased the 
frequency and speed of promotion. 


Employment: Another objective measure of the 
success of career counseling and employment services 
is employee placement and employment rates. The 
following studies evaluated programs that integrated 
an element of career counseling into a larger career 
guidance and services program. For example, several 
studies have evaluated the British New Deal for Young 
People (NDYP), a welfare-to-work strategy that aims 
to help unemployed 18- to 24-year-olds find work and 
increase their employability. According to Reenen 
(2004), NDYP appears to have had a significant 
influence in moving more young people into jobs; 
young unemployed men are about 20% more likely to 
find jobs each month because of NDYP. Dorsett (2006) 
specifically examined the variety of employment 
strategies of NDYP. The first phase of NDYP is the 
Gateway stage, a period of intensive job search under 
the supervision of a caseworker. The second phase 

is the option stage where participants select one of 
four choices: (1) full-time subsidized employment; (2) 
full-time education or training; (3) work placement in 
a voluntary sector organization; or (4) work placement 
in an organization with an explicit environment-focus. 
Employing propensity score matching, Dorsett (2006) 
found that those that entered optional Phase 1 (full- 
time subsidized employment) were most likely to find 
subsequent permanent employment. However, those 
who remained in the Gateway stage (intensive job 
search under the supervision of a caseworker) for 

a prolonged period were in fact more likely to find 
employment than those who entered the remaining 
Phase 2 options (full-time education or training, work 
in the voluntary sector, or work in the environmental 
sector). 


Another study compared the effectiveness of 
employment services versus small-business assistance 
programs in helping the unemployed return to work 
(Rodriguez-Planas, 2007). For the purpose of this study, 
employment services included job clubs, job search 
courses, Counseling, testing, and assessment, whereas 
small business assistance programs provide counseling 
and assistance in developing and implementing 

a business plan, and often included some form of 
financial assistance. Rodriguez-Planas (2007) found 
employment services to be more successful; in 
comparison to participating in small business assistance 
programs, participating in employment services 
increased the likelihood of being employed by 

17.28 percentage points (or 34.18%) for at least 12 
months within the two-year period, and reduced the 
period of unemployment by over 3 months (or 27.48%) 
during the same period. 
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Furthermore, a synthesis of evaluations of job search 
assistance programs including but not limited to job 
referrals, job counseling, skills assessment, job search 
training, resume writing, and interviewing techniques 
found that these programs had some positive effects 
on employment and earnings and were usually cost- 
effective (Dar & Tzannatos, 1999). However, the 
researchers cautiously noted that there appeared 
to be a positive correlation between the likelihood 
of success of job search assistance and local labor 
market conditions: programs that demonstrated 
successful results were situated in locations with 
more favorable economic conditions and lower 
unemployment rates. 


Interestingly, Watts (2009) also found evidence in 
some countries of career counseling favoring general 
education at the expense of vocational education and 
training. Watts (2009) cited examples of studies across 
Australia and the United Kingdom where guidance 
practices served as a barrier to apprenticeship and 
VET participation by either limiting the availability of 
information or discouraging students from pursuing 
such paths. However, this might reflect the tendency 
for career guidance in formal education settings to 

be biased towards programs offered by their own 
institutions. Therefore, Watts (2009) underscored the 
importance of providing additional guidance from 
independent external agencies as offered in Germany 
and Switzerland. Further, for career guidance services 
to provide adequate information on VET programs, this 
requires revamping the training of career guidance 
practitioners as most career practitioners have not 
been trained within the VET system and therefore lack 
the knowledge and expertise to relay to students. 


Limitations and Implications 


It is difficult to draw general conclusions from these 
studies as the evidence is not strictly causal. In 
addition, the relevance and applicability of these 
studies is also tenuous as only a handful of studies 
focus on the student population, and of those only a 
few focus specifically on STEM (2004; Fouad, 1995; Kerr 
& Robinson Kurpius; Stoeger et al., 2013). Based upon 
the available evidence, however, career counseling 
and employment services do seem to be effective 

in increasing subjective outcomes such as career 
initiative, career self-efficacy, and career satisfaction 
(Bernaud et al., 2006; Brennan, 2009; Chang & Feng, 
2014; Collins, 1994; Fouad, 1995; Gong et al., 2014; Kerr 
& Robinson Kurpius, 2004; Pan et al., 20011; Perdix et 
al., 2013; Siebert, 1999; Stoeger et al., 2013; Wu eft al., 
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2013). However, the results of studies evaluating the 
impact of career counseling and employment services 
on increasing objective outcomes such as earnings, 
career advancement, or employment are mixed 
(Chang & Feng, 2014; Collins, 1994; Dar & Tzannatos, 
1999; Dorsett, 2006; Gong et al., 2014; Reenen, 2004; 
Rodriguez-Planas, 2007; Watts, 2009). While most 
researchers found that counseling and employment 
services have a positive impact (Collins, 1994; Dar 

& Tzannatos, 1999; Dorsett, 2006; Gong et al., 2014; 
Reenen, 2004; Rodriguez-Planas, 2007), a few report 
null (Chang & Feng, 2014) or potentially negative 
consequences (Watts, 2009). In addition, Dar and 
Tzannatos (1999) highlight the important role that the 
economic context plays in determining the success 

of career counseling, and Watts (2009) underlines the 
need to overhaul the training of career guidance 
practitioners in order to ensure that they are not only 
knowledgeable of VET career pathways but also aware 
of the need to prioritize and promote VET. 
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Monetary Incentives to 
Students 


Sherihan Khalil 


This section explores the efficacy of monetary 
incentives (such as vouchers, scholarships, or cash 
transfers) to students to encourage attendance, 
reduce dropout rates, and increase enrollment 

in educational programs (particularly technical/ 
vocational education) and also to improve learning 
outcomes and post-secondary school aspirations. It 
includes a description of the literature as a whole, 
the relevant findings associated with this body of 
work, and any limitations or implications of these 
studies. While there have been very few studies 

on the role of monetary incentives in technical 
education specifically, we present the available 
evidence on the impact of monetary incentives in 
education more broadly, assuming that some of our 
findings may be applicable to technical education 
as well. 


Literature Description 


Sixteen studies dealing with the effects of monetary 
incentives to students are synthesized here. The 
following section provides an overview of the 
characteristics of the studies included in this synthesis. 


Study Population. Of the 16 studies that examined 
the impact of monetary incentives to students, most 
targeted students currently attending either primary 
or secondary/vocational school. Further, many of 
these programs targeted very specific subgroups. 
For example, in Colombia, youth from the two lowest 
socioeconomic strata were targeted to receive 
vocational training vouchers (Attanasio et al., 2011), 
and in the Philippines, eligibility requirements for 
TVET scholarships included limits on family income, 
completion of high school, and completion of pre- 
training qualifications (Orbeta & Abrigo, 2013). 


Publication Type. Of these 16 studies, 11 are journal 
articles, and 5 are policy papers or reports. 


Outcomes. Studies examined the impact of monetary 
incentives to students on an array of student outcomes 
including enrollment rates, attendance rates, dropout 
rates, grade progression, test scores, and the likelinood 
of taking a college entrance exam. 


Methodology. Thirteen of the 16 articles utilized a 
quasi-experimental or experimental design. 


Region. In terms of regional scope, 2 papers were 
cross-regional or global in scope, 6 were based in 
Latin America, 5 in Asia, 1 in Africa, 1 in Europe, and 1 
in the United States. 


Findings 


This section synthesizes the impact of monetary 
incentives on student outcomes such as student 
enrollment, attendance, and performance, as well 
as on earnings. The findings of these studies include 
several major topics: the impact of vouchers and 
subsidies; the impact of scholarships and fellowships; 
the impact of scholarships when performance is 
explicitly linked to the incentive; the impact of 
conditional cash transfers (CCTs); and the impact of 
incentives on female attendance and performance. 


Impact of Vouchers and Subsidies: Attanasio et al. 
(2011) evaluated the Jovenes en Accion program in 
Colombia, which provided targeted VET training to 
poor urban youth, most of whom were high school 
dropouts. Among the many courses offered through 
local training institutes were those that trained 
students to become IT specialists, surveyors, or 
accountant assistants, among many other professions 
in administrative and manual occupations. The VET 
training totaled six months equally divided among 
classroom training and on-the-job training. Cash 
transfers to attend the program were administered 
to cover the cost of lunch and transportation, and 
women with children under seven years of age 
received higher cash transfers to accommodate 
childcare expenses. The impact evaluation of the 
program utilized a randomized experimental model 
to conclude that the women in the VET-trained 
treatment group benefited from increased earnings 
(increase of 19.6%) and were more likely to find paid 
employment (0.068 higher probability), mainly from 
jobs in the formal sector. 


Another experimental study in Colombia found 
additional benefits associated with the offer of 
vouchers to vocational schools. By comparing 
voucher lottery winners (who were allowed to select 
their school placement) to voucher lottery losers, 
Bettinger et al. (2010) concluded that the program 
yielded positive impacts on voucher winners’ test 
scores and increased their likelihood of taking a 
college entrance exam, among other positive 
education outcomes. Although the peer groups 
selected by the voucher winners were lower in 
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academic standing than the voucher losers, these 
positive education outcomes remained (Bettinger et 
al., 2010). 


Impact of Scholarships and Fellowships: A randomized 
evaluation of a merit scholarship for girls in two 
districts of Kenya indicated an improvement in test 
scores by 0.19 standard deviations (for the “Intention 
to Treat” samples) (Kremer, Miguel, & Thornton, 2009). 
The study also measured positive externalities within 
schools that offered the scholarship; the authors found 
that although the scholarships were targeted at girls 
alone, the schools that offered the scholarships saw 
an increase in boys’ scores as well by 0.08 standard 
deviations. 


The use of financial incentives for girls in low-income, 
urban neighborhoods of Quetta, Pakistan also showed 
a positive impact on primary school enrollment. A 
randomized impact evaluation of an Urban Girls’ 
Fellowship Program found that the program increased 
enrollments by an average of 33% (Kim, Alderman, 

& Orazema, 1998). The program differs from other 
financial incentives detailed here in that the subsidies 
for girls’ enrollments were directly allocated to 
schools. 

Additionally, Cameron (2000) analyzed the impact 

of a scholarship program offered to primary, lower 
secondary, and upper secondary school students 
during the Indonesian recession of the 1990s, 

and found that these financial incentives were 
positively associated with school attendance. Using 
both regression and propensity score matching to 
establish impact, the author also concluded that the 
scholarship program reduced dropout rates by 24% 
among students in lower secondary schools. 


Further, in the Philippines, 73% of TVET graduates in 
2007 received scholarships through either the Training 
for Work Scholarship Program or the Private Education 
Student Financial Assistance. The groups targeted 

to receive these scholarships were individuals that 
were Classified as underemployed or unemployed. 
Incentives offered to individuals through these 
scholarships included the payment for all training, 

a stipend, a book allowance, career counseling, 

and employment referrals. In their non-experimental 
evaluation of these two scholarship programs, Orbeta 
and Abrigo (2013) show that the scholarships assisted 
in increasing TVET graduation rates yet did not assist 
in improving the employment rate of graduates. The 
study stated that the low employment rate was not a 
function of the scholarship programs but of broader 
sectoral constraints to employment. 
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Impact of Financial Incentives on Student 
Performance when Performance is Incentivized: 
Numerous studies have found that providing a 
financial incentive that is contingent upon academic 
performance can be effective in increasing 
academic achievement or enrollment (Angrist, 
Bettinger, Bloom, King & Kremer, 2002; Angrist, 
Bettinger, & Kremer, 2006; Barrera-Osario & Filmer, 2013; 
Behrman, Parker, Todd & Wolpin, 2012; Fryer, 2011; 
Kremer et al., 2009; Leuven, Oosterbeek, & van der 
Klaauw, 2003). For example, a randomized evaluation 
of merit-based scholarships in Kenya concluded 

that merit-based scholarships were the most cost- 
effective way to boost students’ primary school test 
scores when compared to other incentive schemes 
such as teacher incentive programs or book provision 
programs (Kremer et al., 2009). A Mexico-based 
randomized experiment on the effects of monetary 
incentives to improve mathematics achievement 
compared treatment effects of monetary incentives 
to mathematics exam performance between three 
treatments scenarios: when incentives were given to 
teachers only, students only, and teachers, students 
and administrators. The experiment concluded that 
largest gain in test scores were seen in schools in 
which students, teachers, and administrators received 
incentives, with gains also seen in schools in which 
only students received incentives; no significant gains, 
however, were seen in schools with teachers only 
receiving benefits (Behrman et al., 2012). 


Further, a Cambodia-based RCT study comparing 
primary school and lower secondary school 
scholarships based on poverty level versus scholarships 
based on merit (higher test scores), found increased 
enrollments from both types of scholarships; however 
only the merit scholarships were associated with 
increased test scores (Barrera-Osario & Filmer, 2013). 
Others have found incentive effects of smaller 
magnitude, as well as effects that are inconsistent 
across subjects or grades. For example, Fryer (2011) 
conducted a United States-based RCT and found that 
paying second grade students to read significantly 
improves students’ reading achievement but has a 
detrimental impact on English language learners, and 
no impact on 9th graders. 


Further, Angrist et al. (2002) conducted a study of 
Colombia's use of lotteries to distribute vouchers, 
which partially covered the cost of private secondary 
school for students who maintained satisfactory 
academic achievement. They found that voucher 
winners were not only about 10 percentage points 
more likely to finish 8th grade, but also scored 0.2 
standard deviations higher on math and reading 
scores. Another study of Colombia's program also 
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found that the vouchers increased high school 
graduation rates by 15-20% (Angrist et al., 2006). 

At the tertiary school level, Leuven et al. (2003) 
conducted a randomized field experiment among 
first-year economics and business students at the 
University of Amsterdam to examine the effectiveness 
of financial incentives on student achievement 

when performance is incentivized. First-year students 
could earn financial rewards for passing first-year 
requirements in one year. They found that the 
financial incentives increased the achievement of 
high-ability students, but decreased the achievement 
of low-ability students. 


Impact of CCTs: Though the literature on financial 
incentives in general may portray a positive to mixed 
review on the impact of financial incentives on 
student attendance, enrollment, and performance, 
the literature on conditional cash transfers in particular 
is largely positive in regards to attendance and 
enrollment (though not performance). The Rand 
Corporation's meta-analysis of 42 impact evaluations 
conducted in 15 developing countries in Latin 
America and Asia found that CCTs had a statistically 
significant impact on attendance for all age groups 
with the average effect much larger for secondary 
school students (a 12 percentage point impact for 
secondary school students versus a 3 percentage 
point increase for primary school students) (Saavedra 
& Garcia, 2012). Secondly, the authors found that 
payment amount and payment frequency were 
critical factors influencing CCT impact: the more 
generous the CCT, and the more frequent the 
payments, the larger the impact. Lastly, authors found 
that pay linked to achievement also increases impact. 


In addition, Baird, Ferreira, Ozler and Woolcook (2013) 
examined 11 studies on the use of both CCTs and 

UCT (unconditional cash transfers) and also found 
positive and significant effects of these programs on 
enrollment. Specific examples of these studies include 
an impact evaluation of the 1997 Progresa program 
in Mexico, which provided a positive appraisal of the 
provision of financial incentives to families, contingent 
upon children’s regular school attendance. Progresa, 
which funds both primary and secondary school 
students, resulted in higher enrollments, quicker grade 
progression rates, and lower dropout rates (Behrman, 
Sengupta, & Todd, 2005). Further, in Asia, Mo et al. 
(2013) conducted an RCT examining the role of CCTs 
in reducing drop out rates in rural China; they found 
that the dropout rate in the CCT group (5.3%) was 
eight percentage points lower than in the control 
group (13.3%). 


Impact of Financial Incentives on Females: 
Improvements in female attendance and/or 
performance are apparent in both programs that 
specifically target girls and programs that target both 
genders but contain gender-sensitive components. 
An example of the latter includes CCT programs 

such as Progresa in Mexico and a VET training to 
poor urban youth, Jovenes en Accion, in Colombia 
that provides financial incentives to both males and 
females for training and attendance. Although both 
programs target overall populations, their incentives 
structures are not gender neutral. For example, 
Progresa provides girls with higher financial incentives 
than boys (Behrman et al., 2005), and Jovenes en 
Accion provides women with children under the 

age of seven with higher cash transfers for childcare 
costs (Attanasio et al., 2011). With this set up, the 
impact on women's earnings and enrollment was 
significantly higher than boys or men (Behrman et 

al., 2005; Attanasio et al., 2011). Finally, in terms of 
female-specific programs, multiple impact evaluations 
indicated that these targeted incentive programs 
have positive effects on female school enrollment and 
attendance; examples include Cambodia (Filmer & 
Schady 2008), Bangladesh (Raynor & Wesson, 2006), 
Pakistan (Kim et al., 1998), and Kenya (Kremer et all., 
2009). 


Limitations and Implications 


Based on the studies included in this synthesis, it 
appears that on average, financial incentives can 
increase attendance and enrollment (Kim et al., 
1998; Cameron, 2000; Behrman et al., 2005; Filmer & 
Schady, 2008; Kremer et al., 2009; Raynor & Wesson, 
2006; Saavedra & Garcia, 2012; Barrera-Osario & 
Filmer, 2013; Baird et al., 2013), grade progression 
(Cameron, 2000; Angrist et al., 2002; Behrman et al., 
2005), graduation rates (Orbeta & Abrigo et al., 2013; 
Angrist et al, 2006), academic performance (Angrist 
et al., 2002; Leuven et al., 2003; Kremer et al., 2009; 
Bettinger et al., 2010; Fryer, 2011; Barrera-Osario & 
Filmer, 2013), and can also reduce dropout rates 
(Behrman et al., 2005; Mo et al., 2013). It is important 
to note that many of these programs targeted specific 
subgroups. Therefore, the effectiveness of financial 
incentives may not only vary by characteristics of 
the incentive—the amount and frequency—but also 
by characteristics of the recipients—school level, 
academic ability, English language status, and 
gender. 


It is important to note that although most studies in 
this sample seem to report that financial incentives 
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have a positive effect on student outcomes, only a 
few of these studies are specific to VET. Of the studies 
exploring the impact of vouchers and subsidies only 
two were VET-specific (Bettinger et al., 2010; Attanasio 
et al., 2011), and of those exploring the impact of 
scholarships or fellowships, only one was VET-specific 
(Orbeta & Abrigo, 2013). Thus while these incentives 
certainly have the potential to improve educational 
outcomes in the field of technical education as well, 
the current state of the evidence-base does not yet 
allow for such definitive Conclusions. 
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Conclusion 


This literature review provides a broad 
overview of the impact of various strategies 
for growing technicians and skilled workers 
employed by various national governments, 
non-governmental organizations, technical 
schools, and industries. Although this 
synthesis is to an extent limited by the 
availability of rigorous impact evaluations, 
the studies considered here do still offer 
policy recommendations for each of the 
five strategies used to address the need for 
skills development: vocational education, 
apprenticeships, cooperative education, 
career academies, and e-learning programs. 


Vocational education. Based on the impact evaluations 
considered in this literature review, there are several 
recommendations we propose for vocational 
education programs. First, the literature suggests that 
the tighter the link between vocational education 
programs and industries or firms, the more successful 
the intervention. For labor market training programs, 
for example, industries could use company-based 
vocational training programs as a means of recruiting 
future employees. Second, high quality labor market 
training institutions had a much greater impact on 
students, thus any government or private sector- 
funded vocational education initiative should employ 
a competitive bidding process in choosing vocational 
programs to support. This process should consider factors 
including but not limited to the legal status (formality) 
of the program; the level of human resources and 
infrastructure or resources supporting the program; the 
appropriateness of coursework/curricular structure given 
the labor demand; and the expenditures per trainee. 
Third, if the organization in charge of implementing the 
vocational education program also functioned as a 
job placement and counseling organization, this could 
be an efficient way to track students and ensure their 
success. Lastly, policymakers should consider focusing 
on untapped markets. Preliminary evidence suggests 
that some private vocational schools may be more 
equipped to adjust to changing market demands than 
their public counterparts. Moreover, programs should 
target women to participate since studies suggest 

that women consistently seem to benefit more from 
vocational training than their male counterparts but 
are also less likely to sign up for vocational training 
(particularly the type of vocational training with high 
returns such as training in traditionally male-dominated 
trades). 


Strategies for Strengthening the Technical Workforce 


Apprenticeships. Apart from a randomized study of 

a national apprenticeship program in Ghana which 

is currently underway (Lehrer, Mbiti, & McCasland, 
2014), the rigorous efficacy literature on apprenticeship 
programs is limited to Europe, making it difficult to draw 
inferences from these studies to other contexts. However, 
within the European system, there are several summative 
findings. Apprenticeship trainings were linked to positive 
income returns in the short run, though in the long run 
these returns may decline; shorter unemployment spells 
(compared to unskilled workers); and an increased 
probability of finding immediate employment (post- 
apprenticeship). However, when compared to 
individuals who have completed secondary education 
or VET, the results for apprenticeship training were 

more mixed. There was also variation in the returns 

of apprenticeships based on student characteristics. 

For example, returns were higher for men (given 

that women were more likely to have completed 
apprenticeships in more female-dominated, low- 

wage sectors), for students with high quality education 
credentials, and for those that complete more “intense” 
or prestigious apprenticeships. 


Cooperative education. Only a few studies of 
cooperative education programs used experimental 
or quasi-experimental methods, so any differences 
between participants and non-participants may not 
be fully attributable to these cooperative education 
programs alone. Additionally, several of these studies 
employed only self-reported data. Nonetheless, based 
on the studies included in this literature review, it seems 
that cooperative education appears to have high 
potential for connecting school experience with the 
workplace and for keeping students in school. A number 
of the studies found that students who completed 
cooperative education programs received higher 
starting salaries and spent less time finding a first job 
placement compared to non-cooperative education 
students. Furthermore, student responses to program 
surveys offered several recommendations for improving 
cooperative education programs, such as offering 
guidance about industries in which students could be 
placed for work, providing continuous mentoring by 
adults while working, and implementing high-quality 
transition programs. 


Career academies. While several studies found career 
academies to be associated with positive education 
outcomes (in terms of school attendance, credits 
earned toward high school graduation, grade point 
average, and retention through high school), the most 
rigorous study of career academies (a randomized 
trial), conducted by MDRC, found a great degree of 
heterogeneity in impact, depending on the particular 
school and program. Further, these studies indicated 
that though career academy enrollment was associated 
with higher graduation rates, the impact on student 
dropout rates was mixed, except in the case of high-risk 
students where the program seems to have had higher 
positive impacts overall. And while the results on tertiary 


education aspirations, admissions, and enrollment are 
more mixed, career academy students were provided 
with more labor market preparation and also tended 
to earn more than non-academy students in the labor 
market, with impacts slightly higher among young men. 


Workforce training through e-learning. Overall the 
research and policy literature on e-learning training 
indicate that the success of e-learning programs in 
post-secondary populations is not significantly different 
from face-to-face programs. However, certain studies 
indicate that these programs may be less effective 

for low-income students, while at the same time more 
effective for students performing at both particularly 
high and low levels, due to the ability of these programs 
to target the particular learning needs of students The 
evidence also indicates that blended learning, which is 
instructor-led and collaborative, is more effective than 
fully online, independent learning, particularly when 
blended learning is accompanied by extensive practice 
and instructor feedback. There is also a substantial 
amount of policy literature which highlights the potential 
for e-learning to completely transform the technical 
workforce. However, this literature also highlights the 
need for ICT skills development in the student and 
teacher population overall as a precursor to workforce 
e-learning, as well as the need for investments in training 
time (among employers) and an e-learning infrastructure 
at both the country and industry-level. 


In addition to the strategies outlined above to strengthen 
preparation for technical employment, the overall 
access people have to information and incentives 
related to participating in a technical program also 
needs to be improved. While only a few studies address 
the efficacy of strategies used to increase accessibility to 
information and incentives—informational interventions, 
career counseling coupled with employment services, 
and monetary incentives—the available research still 
provide several noteworthy suggestions and underscore 
areas that require further investigation. 


Information interventions. The limited sample of impact 
evaluations in this review found that informing students 
of the returns associated with various levels of education 
was effective in influencing school choice. Further, the 
more individuals’ misperceptions of these returns differ 
from reality, the more impactful an intervention can 
be. For example, in countries where individuals had 
overly optimistic perceptions of the rate of return to 
vocational education, informational interventions had 
no statistically significant impact on student demand 
for vocational education. Contrastingly, where the 
rate of return to education was perceived as lower 
than its real value, informational interventions had a 
positive impact on schooling attainment. In addition 
to ensuring that people have an accurate perception 
of the returns of vocational education, counseling and 
guidance about the options open to adults (as well as 
the returns on those options) and assistance with filling 
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out complicated forms, may generate an even larger 
impact on individual choices. 


Career counseling and employment services. The 
limited available evidence suggests that career 
counseling and employment services are effective in 
increasing subjective outcomes such as career initiative, 
career self-efficacy, career commitment, and career 
satisfaction. However, the studies revealed mixed results 
on the association between career counseling and 
increasing objective outcomes such as earnings, career 
advancement, or employment; while most researchers 
found that counseling and employment services have 
a positive impact on these factors, a few reported null 
findings, which were partially driven by local economic 
conditions. 


Monetary incentives to students. According to the 
studies included in this synthesis, on average, financial 
incentives increase attendance and enrollment, 

grade progression, graduation rates, and academic 
performance, and can also reduce dropout rates. 
However, since many incentive programs target 
specific groups, their effectiveness may vary both 

by characteristics of the incentive (the amount and 
frequency) and by characteristics of the recipients 
(school level, academic ability, English language status, 
and gender). Furthermore, while financial incentives 
certainly have the potential to improve education 
outcomes, only a few of these studies were specific to 
vocation or technical education, thus we cannot yet 
draw definitive conclusions about the impact of these 
incentives on VET enrollment, subsequent education, or 
labor market outcomes. 


The research evidence synthesized here indicates that 
there is overall a high potential to improve the technical 
skills of the workforce through vocational education, 
apprenticeships, cooperative education, career 
academies, and e-learning programs. However, the 
extent to which each of these strategies is successful 
varies based on a variety of factors, including the 
country context, the duration/completion of the 
program, the availability of support systems, the 
background of participants, etc. Equally important to this 
discussion is the accessibility of information about and 
incentives for partaking in the aforementioned programs, 
whether it is through information interventions, career 
counseling and employment services, and/or monetary 
incentives to students. Similar to the program strategies, 
the effectiveness of each of these strategies to increase 
accessibility also varies by context. Nonetheless, the 
available evidence suggests that policies and practices 
that promote skill-develooment programs and the 
accessibility of information and incentives would 
encourage the growth of the technical workforce. 
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APPENDIX A. Occupational Information Systems 


Year Study title/Author Summary of 
findings 

O*NET http://www.onetcenter.org/overview.html USA 

Asia Power http://asiapower.in Asia 

National Careers Service https://nationalcareersservice.direct.gov.uk/Pages/Home.aspx UK 

Northern Ireland Direct http://www. nidirect.gov.uk/careers Ireland 

International Cenire for http://www.derby.ac.uk/research/icegs/ UK 


Guidance Studies 


National Careers Council https://www.gov.uk/government/groups/the-national-careers- UK 
council 
European Lifelong Guidance http://www.elgpn.eu EU 


Policy Network 


OECD http://www.oecd.org/education/innovation-education/ N/A 
careerguidancepolicyreviewhomepage.htm 


Careers New Zealand http://www.careers.govt.nz New 
Zealand 
Korean Research Institute for http://eng.krivet.re.kr/eu/index.jsp Korea 


Vocational Education & Teaching 


Asia Pacific Career Development http://www. asiapacificcda.org Asia 
Association 

Canadian Career Development http://www.ccdf.ca/ccdf/ Canada 
Foundation 

International Association for http://www.iaevg.org/iaevg/index.cfm#?lang=2 N/A 
Educational and Vocational 

Guidance 

Job Guide http://www.jobguide.education.gov.au Australia 
Australian Apprenticeships http://www.australianapprenticeships.gov.au Australia 
National Cenire for Vocational http://www.ncver.edu.au Australia 


Education Research 


Australian Government Dept. of http://training.gov.au Australia 
Education and Training 


Job Services Australia http://employment.gov.au/job-services-australia-jsa Australia 


60) Consortium for Policy Research in Education | cpre.org 


APPENDIX B. Policy Guidance Literature 


In addition to the efficacy literature, which makes up the bulk of this literature review, a number of policy-relevant 
reports, manuals, and books have been identified through our systematic searches. Here, we provide a brief 
description of eight of the most recent, relevant, and high-quality “policy guidance papers” as tools that could be 
used by policymakers moving forward. 


Active labor market policies by topic: 


e The OECD's “Learning for Jobs” (Field et al., 2010) is a synthesis report of reviews of vocational education and 
training and provides policy guidance for practitioners on active labor market strategies (vocational education, 
career guidance, workplace learning, tools to support the system, etc.). The guidance is directed at OECD 
countries but may have implications for other regions. 


e From the Asian Development Bank (Maclean et al., 2013), the book, Skills Development for Inclusive and 
Sustainable Growth in Developing Asia-Pacific, is a collection of views and insights from policymakers, 
practitioners, and leading experts on skills development for “inclusive and sustainable growth.” The book 
examines current strategies in TVET in the Asia and Pacific region, with special attention to the “link between 
technical and vocational education and training and the imperatives of greening economies.” The authors also 
propose an international framework (across the Asia and Pacific region) for policymakers to consider moving 
forward. 


e The International Handbook of Career Guidance (Athanasou & Esbroeck, 2008) is a reference book which 
provides a comprehensive view of career guidance policies internationally, with particular emphasis on 
vocational guidance. The handbook covers the following topics: educational and vocational guidance in 
a social context; theoretical foundations; educational and vocational guidance in practice; specific target 
groups; testing and assessment; and evaluation. 


e The Review of Vocational Education: The Wolfe Report (Wolfe, 2011), focuses on vocational education for 14- to 
19-year-olds in England, and summarizes a wealth of qualitative evidence that sheds light on ways to promote 
the successful progression of this population into the labor market, as well as higher education and training. The 
report provides policy recommendations that take special account of England's current financial situation. 


Active labor market policies by country (case studies): 


e The OECD's “Skills Develooment Pathways in Asia” (Maclean, Jagannathan, & Sarvi , 2012) is a report that 
provides a series of case studies across 15 Asian nations, which detail the strategies for skills development in 
each country. These case studies are summaries of discussions held at the annual meetings of the Employment 
and Skills Strategies in Southeast Asia initiative (ESSSA) in Tokyo (2010) and Shanghai (2011) in relation to policies 
and programs for skills development approaches in Asian countries. Countries covered include: Australia, 
Cambodia, China, Hong Kong, China, India, Japan, Korea, Malaysia, Mongolia, Nepal, Pakistan, the Philippines, 
Singapore, Thailand, and Vietnam. 


Guidance on the use of ICTs in technical and vocational training: 


e UNESCO's (Chinien, 2003) report on “The use of ICTs in Technical and Vocational Education and Training,” 
reviews the use of ICTs for administrative purposes; Communication, teaching and learning; curriculum 
development and assessment; career education and guidance; labor market information; job placement; and 
systems control. Through survey and case study means, the authors (1) provide an overview of the various types 
of programs found under each of the above topics and (2) provide case study evidence of the ways in which 
countries are using ICTs for TVET. 


e UNESCO's “ICT Application in Technical and Vocational Education and Training: Specialized training course” 
(Chinien, 2005), provides participants with the opportunity to “acquire and develop the knowledge and 
practical skills necessary to establish, manage, and appraise |CT-mediated teaching and learning in TVET.” 
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This course was designed specifically for policymakers and managers/administrators of TVET-related programs 
and trainings, and includes three models: Implementation of |CT-mediated teaching and learning in TVET; ICT- 
mediated teaching in TVET; and ICT-mediated learning in TVET. Each unit of the three-module course includes 
instructional objectives, relevant training materials for each objective, and corresponding application exercises. 


Conducting research on TVET policies: 


e The Handbook of Technical and Vocational Education and Training Research from UNEVOC focuses on the 
research methods used to evaluate labor market strategies and vocational training in particular (Rauner 
& Rupert, 2009). The authors provide a review of the methods, including a sample of case studies used to 
illustrate different approaches to TVET research. The research methods presented include interview-based and 
observational methods, as well as experimental and process evaluations. This may be a useful resource in setting 
up an evaluation of the Thai government’s new labor market policies. 


e The book, Competence Development and Assessment in TVET (COMET): Theoretical Framework and Empirical 
Results, also from UNEVOC, uses a case study to test a methodology for measuring vocational competence, 
which could potentially be employed internationally. The authors demonstrate that the “methodology can 
be applied to designing and evaluating vocational education and training processes,” making the material 
relevant to VET teachers and trainers as well as academics. 
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APPENDIX C. Literature Search Strategy 


The literature search strategy is described below for each topic: 


Vocational Education 


Fifteen studies of the impact of vocational education on youth were identified for inclusion in this synthesis. 
These studies described were identified through (1) a review of a recently published meta-analysis on 
vocational education and training in low and middle income countries ; (2) citation crawling of meta- 
analysis and highly cited studies; (3) a search of the Impact Evaluation (3ie) database and the Youth 
Employment Inventory ; and (4) primary searches of the following databases from 06/01/2014-08/15/2014: 
Education Research Complete, Business Source Complete, ERIC, Education Full Text, Social Sciences Full Text, 
and Econ Lit with Full Text, as well as Google Scholar. Search terms included: “workforce training,” “vocational 
programs,” “vocational education,” “vocational school,” “vocational training,” “trade school,” “school-to- 
work,” “school-to-career,” “VET,” “TVET,” “tech prep,” “active labor market policies,” “active labor market 
programs,” “STEM technicians,” and “dual vocational.” 


6b 


Apprenticeships 


Twenty-seven efficacy studies regarding the impact of apprenticeships on youth outcomes were identified 
for inclusion in this synthesis. In addition, one systematic review examining the impact of apprenticeships 
on life outcomes (in Europe) was also included in this review. These studies described were identified in 
two main ways: (1) through citation crawling of the meta-analyses and highly cited studies, including a 
recent systematic review (Torchio, 2013) and (2) through primary searches of the following databases from 
08/01/2014-08/15/2014: Education Research Complete, Business Source Complete, ERIC, Education Full 
Text, Social Sciences Full Text, and EconLit with Full Text, as well as Google Scholar. Search terms included: 
“apprenticeship” OR “active labor market policies” OR “active labor market programs” OR “workplace 
learning” OR “workforce development” OR “career internship” OR “learning-by-doing” OR “on-the-job 
training.” 


Cooperative Education 


Twenty-three studies regarding the impact of cooperative education were identified for inclusion in this 
synthesis. Based on a systematic review examining cooperative education in EBSCO host with Education 
Research Complete, Business Source Complete, ERIC, Education Full Text, Social Sciences Full Text, and Econ 
Lit with Full Text, covering the years 2000 2014, the most relevant impact studies were identified. Upon the 
reading of these studies, influential studies prior to 2000 were also added to the analysis. 

Career Academies 


Seventeen studies are included in the analysis. The search process for studies examining the impact of career 
academies on students included: 


a. A word search with “Career Academies” and “Career Academies and Education work relationship” to find 
literature in the ERIC database published between 2000-2014 (728 hits). Among them, 385 scholarly articles 
and 50 evaluative reports were identified as potentially relevant. The abstract of each publication was 
then reviewed to determine the rigor of its research design. Seventeen publications were selected for the 
literature review. 


b. Asearch of education databases and labor economy databases via EBSCOhost. Five relevant databases 
were included for the search: EconLit with Full Text; Education Full Text (H.W. Wilson); Education Research 
Complete; ERIC; and PsycINFO. The same search terms were used: “Career Academies” or “Career 
Academies and Education work relationship” within the abstract, published between 2000-2014. Two 
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hundred seventeen articles were identified, and the abstract of each one was reviewed to identify its rigor. 
Sixteen additional publications were added to the literature review. 


c. Asearch of the website of Association for Career and Technical Education, which includes the clearing 
house on Career Academies and Small Learning Communities. Two additional publications that discuss the 
CTE framework were added to the pool. 


d. In addition to the literature found through systematic database research, three studies conducted prior to 
2000 were identified as important from examining study references and were included (Stern, et al., 1989; 
Linnehan, 1996; Kimpel, 1997). 


e. In all, 39 articles and evaluative reports were reviewed for this section of the review, and findings from the 
most relevant and rigorous 17 are included in the analysis. 


Workforce Training Through E-learning 


Twenty efficacy studies in the field of e-learning for post-secondary young adults/adults were identified for 
inclusion. In addition, two meta-analyses and 30 high-quality, non-experimental case studies, journal articles, 
and policy reports were included. 


These studies were identified through a systematic search of the literature, including searches of the 
following databases conducted from 01/10/2014-01/17/2014: Education Research Complete, Business Source 
Complete, ERIC, Education Full Text, Social Sciences Full Text, and EconLit with Full Text. The search terms used 
for this portion of the literature review were as follows (concatenated): 


a. E-learning related: “online” OR “on-line” OR “e-learning” OR “distance learning” OR “distance education” 
OR “ICT” OR “IT” OR “information technology” OR “computer assisted” OR “computer-assisted” OR 
“computer based training” OR “computer-based instruction” OR “virtual” OR “computer-mediated” OR 
“web-based” OR “electronic learning” OR “technology-enabled” OR “communications technology” OR 
“ICT for TVET’ or “ICT for VET” 


b. Education/Training related: “education” OR “learning” OR “training” OR “job training” OR “instruction” OR 
“program” OR “skill*” OR “degree” OR “certificate” OR “credential” OR “vocation*” OR “school” OR “STEM” 


c. Vocational/workforce related: “vocational education” “workforce development” OR “work” OR 
“workforce” OR “job” OR “industry” OR “industries” OR “on-the-job” OR “enterprise*” OR “economy” OR 
“profession” OR “manpower” OR “skill training” OR “human resource development” 


In addition, citation crawling of the references in the studies identified was used, and searches of the World 
Bank, UNESCO, ILO, and OECD websites were also conducted. 


Information Interventions 


Six impact evaluations dealing with the effects of information provision on educational choices were 
included in this synthesis, as well as one descriptive study of student perceptions of educational opportunities 
(seven studies in total). These studies were identified in two main ways: (1) through primary searches of 
academic databases: Education Research Complete, Business Source Complete, ERIC, Education Full Text, 
Social Sciences Full Text, and EconLit with Full Text and searches conducted from 08/01/2014-08/18/2014 and 
(2) through “citation crawling” of the references listed in the studies found below. Search terms for these 
informational studies included the terms: “educational information,” OR “information interventions,” OR 
“educational services,” OR “educational guidance,” OR “returns to education,” OR “student perceptions 
and education,” OR “student expectations and education.” 
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Career Counseling 


Eighteen efficacy studies regarding the impact of career counseling on youth and current employees and 
professionals were identified for inclusion in this synthesis. Studies detailed below were identified in three 
ways: (1) through a recently published meta-analysis on career counseling and mentoring, (2) through 
“citation crawling” of the studies found in the references below, and (3) through primary searches of 
academic databases (including EBSCOhost, ERIC and ProQuest), journals, and Google Scholar. Searches 
were conducted from 7/24/2014 8/14/2014. The key search terms include: “career,” “counseling,” 


“guidance,” “mentoring,” “math,” “science,” “STEM,” “effect,” “evaluation,” “impact,” “earning, 
and “salary.” 
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Access to Industry Credentials 


There was no efficacy literature that looked explicitly at the impact of access to industry credentials on 
student outcomes. A search of the literature was conducted, including searches of the following databases: 
Education Research Complete, Business Source Complete, ERIC, Education Full Text, Social Sciences Full Text, 
and EconLit with Full Text. Instead example s of current credentialing programs and systems (found through 
articles, reports, and websites) were located through searches on Google. Key search terms included: 


“certification system,” “credential system,” “training system,” “industry,” “government,” “STEM,” “technical,” 
and “vocational.” 
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Monetary Incentives for Students 


Preliminary research findings using financial incentives concepts paired with key words such as “non-formal 
education,” “science and technology,” “adult education,” and “Asia” yielded few results. After exhaustive 
attempts using different key word combinations to find high-quality literature specific to this, it became 
apparent that literature on financial incentives within the non-formal STEM sector is non-existent. 

The next phase of the search used Proquest, Ebscohost, and ERIC searches and included key word 
combinations with vocational education on one end and incentive concepts on the other. Incentive 
concept key words used to search the databases included words such as “incentives,” “pay students,” 
“tuition waiver,” “financial aid,” “training subsidy,” “loans,” “disincentives,” “vouchers,” “demand-side 
financing,” and “conditional cash transfers.” Due to limited results in preliminary research, the years searched 
were left open ended, but a parameter set for inclusion into the review included post 1997 studies. 


Finally, citation tracking, Google, and specific organization searches were used to extract papers more 
relevant to the focus of this review. Specially, the search sought to find impact evaluations of incentive 
employing projects. Research published, overseen, or assisted by the World Bank, the OECD, and the Asian 
Development Bank was examined. The papers included in this review maintained standards of a sound 
methodology, with preference given to studies utilizing experimental or quasi-experimental design. In totality, 
these search terms and strategies yielded a total of 28 studies included in this synthesis. 


Please note that throughout the course of this literature review non-evaluative literature was also uncovered 
and entered into a separate spreadsheet with the names of organizations and programs that use incentives 
to assist in learning goals. Details on what the incentive is and to whom it is given are provided. 


Incentives for Industries 


The first phase of the search for literature relevant to industry incentives (incentives for employers to provide 
on the job training or TVET), included search terms and synonyms of terms such as “employer benefits,” “on 
the job training,” “training stipend,” and “subsidized training.” The results did not yield high quality papers 
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relevant to the goal of this paper, however, the search overlapped with findings on industry-recognized 
credentials as incentives. 

The next phase of the search, searching within the framework of active labor market policy, led to more 
relevant and high-quality literature. Examples of search terms in this phase include “Asia,” “active labor 
market policy,” “wage subsidies,” “employer tax incentive,” and “tax credit.” Databases searched include 
Ebscohost, ERIC, and Proquest. The Google search engine was also used to retrieve information from 
international organizations. The years were left open-ended to offer leads on expert authors and institutions 
regardless of year. 

Of a total of fifteen papers found, only eight have been included into this synthesis. Parameters for inclusion 
and exclusion are detailed below: 


a. Peer reviewed journals or academic institution publications received priority for inclusion. 


b. Papers published by institutions whose findings seemed tied to the interests of the research client or subjects 
were excluded. 


c. Policy and working papers that established sound findings relevant to the subject were included. 


d. Papers that discussed employer incentives within an economic framework that did not provide direct 
evidence on the impact of incentives on motivating employers were included. 


e. Papers published through institutions such as the Bonn-based Institute for the Study of Labor, the World Bank, 
and the OECD were included. 
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